00010000:00010000:2.0:1788572076.372869:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572076.372877:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f800 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584176640:12345-192.168.202.157@tcp:104: Request processed in 142us (479us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572076.372886:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9468 00000080:00200000:2.0:1788572076.430161:0:156950:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.430204:0:156950:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.430231:0:156950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.430252:0:156950:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.430261:0:156950:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.430270:0:156950:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.430275:0:156950:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.430283:0:156950:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730180bb58 00010000:00010000:2.0:1788572076.430288:0:156950:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.430298:0:156950:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.430315:0:156950:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.430325:0:156950:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=29 00000080:00200000:2.0:1788572076.430331:0:156950:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.430346:0:156950:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=13 00000080:00200000:2.0:1788572076.430350:0:156950:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572076.430364:0:156950:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572076.430428:0:156950:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb0db lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.430438:0:156950:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb0db lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156950 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.430451:0:156950:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb0db lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156950 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572076.430496:0:156950:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156950:1875444569441024:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572076.431415:0:156950:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fa80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572076.551331:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00010000:00010000:3.0:1788572076.561862:0:15200:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71200/0x8364c713e25bb0fe lrc: 3/0,0 mode: PR/PR res: [0x240002728:0x42:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x265cfe11109bcae2 expref: -99 pid: 156951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.562507:0:15200:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71200/0x8364c713e25bb0fe lrc: 2/0,0 mode: --/PR res: [0x240002728:0x42:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bcae2 expref: -99 pid: 156951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.562517:0:15200:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572076.562547:0:15200:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71200/0x8364c713e25bb0fe lrc: 1/0,0 mode: --/PR res: [0x240002728:0x42:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bcae2 expref: -99 pid: 156951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.562555:0:15200:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71200/0x8364c713e25bb0fe lrc: 0/0,0 mode: --/PR res: [0x240002728:0x42:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bcae2 expref: -99 pid: 156951 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.562580:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572076.562582:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572076.563738:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572076.563779:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569442432:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572076.581328:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569442432:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572076.622309:0:156951:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156951:1875444569442176:192.168.202.157@tcp:36:rmdir.0 00000080:00200000:3.0:1788572076.622385:0:156951:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x2037:0x0](ffff920a50d3a1c8) 00000080:00200000:2.0:1788572076.645960:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.646007:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.646033:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.646054:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.646065:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.646074:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.646079:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.646088:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730168fb58 00010000:00010000:2.0:1788572076.646092:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.646103:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.646126:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.646135:0:156957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=29 00000080:00200000:2.0:1788572076.646142:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.646155:0:156957:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=13 00000080:00200000:2.0:1788572076.646159:0:156957:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572076.646173:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572076.646231:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.646238:0:156957:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.646248:0:156957:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572076.646282:0:156957:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442560:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572076.655165:0:156957:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.664907:0:156957:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442560:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572076.664940:0:156957:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.664957:0:156957:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.664965:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.664973:0:156957:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.664985:0:156957:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.664992:0:156957:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816200 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572076.664997:0:156957:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665006:0:156957:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665014:0:156957:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 5/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665022:0:156957:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572076.665031:0:156957:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665039:0:156957:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb105 lrc: 0/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.665063:0:156957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a52893b80 x1875444569442560/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572076.665201:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572076.665240:0:156957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:3.0:1788572076.665331:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572076.665352:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572076.665363:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.665375:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.665383:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572076.665389:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572076.665392:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:3.0:1788572076.665397:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730168fbe8 00010000:00010000:3.0:1788572076.665401:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.665410:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.665424:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572076.665433:0:156957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=16408 00000080:00200000:3.0:1788572076.665450:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:3.0:1788572076.665464:0:156957:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=16392 00000080:00200000:3.0:1788572076.665468:0:156957:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:3.0:1788572076.665481:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572076.665530:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665538:0:156957:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.665548:0:156957:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572076.665574:0:156957:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442688:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572076.665923:0:156957:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822f180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.684258:0:156957:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442688:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572076.684288:0:156957:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684307:0:156957:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684315:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684322:0:156957:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684332:0:156957:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684339:0:156957:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec6c00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572076.684344:0:156957:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684352:0:156957:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684361:0:156957:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 5/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684370:0:156957:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572076.684379:0:156957:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684386:0:156957:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb10c lrc: 0/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.684407:0:156957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a51b3e300 x1875444569442688/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572076.684426:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572076.684470:0:156957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:3.0:1788572076.684568:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572076.684637:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572076.684649:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.684663:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.684670:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572076.684676:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572076.684679:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:3.0:1788572076.684684:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730168fb58 00010000:00010000:3.0:1788572076.684688:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.684696:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.684710:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572076.684718:0:156957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=13 00000080:00200008:3.0:1788572076.684728:0:156957:0:(file.c:6296:ll_getattr_dentry()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572076.684733:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908),name=d71a.replay-single 00000002:00010000:3.0:1788572076.684745:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x1b8b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572076.684750:0:156957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00010000:00010000:3.0:1788572076.684794:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684802:0:156957:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.684813:0:156957:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156957 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572076.684838:0:156957:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442816:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572076.684923:0:156957:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822f180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.697736:0:156957:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569442816:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572076.697774:0:156957:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697797:0:156957:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697807:0:156957:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697820:0:156957:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697839:0:156957:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697854:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697864:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697873:0:156957:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.697896:0:156957:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3e300 x1875444569442816/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572076.697912:0:156957:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.697924:0:156957:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/7051 (0 0) 00000080:00200000:3.0:1788572076.697932:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572076.697966:0:156957:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x1b8b:0x0], updating i_size 4096 00000080:00010000:3.0:1788572076.697974:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) for lock 0x8364c713e25bb113 00000080:00010000:3.0:1788572076.697984:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00000080:00010000:3.0:1788572076.697987:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb113 from it ffffaf730168fcb8 00010000:00010000:3.0:1788572076.697991:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.698007:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572076.699986:0:156957:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572076.699999:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.700029:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.700051:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.700081:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.700089:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.700099:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.700103:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.700113:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730168fcb8 00010000:00010000:2.0:1788572076.700116:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.700127:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.700341:0:156957:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.700352:0:156957:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.700360:0:156957:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.700370:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.700376:0:156957:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.700380:0:156957:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.700383:0:156957:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.700387:0:156957:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf730168fbf8 00010000:00010000:2.0:1788572076.700390:0:156957:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.700398:0:156957:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.700407:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.700417:0:156957:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=267 00000080:00200000:2.0:1788572076.700425:0:156957:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 44 00000080:00200008:2.0:1788572076.700434:0:156957:0:(file.c:1090:ll_file_open()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00000080:00200080:2.0:1788572076.700473:0:156957:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) cmd=c03066f0 arg=7fffe3cf7020 00000080:00200000:2.0:1788572076.700489:0:156957:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) name=test stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572076.700531:0:156957:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572076.700547:0:156957:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6400/0x8364c713e25aecaa lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d8 expref: -99 pid: 153007 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.700754:0:156957:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572076.700763:0:156957:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572076.700766:0:156957:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7e00/0x8364c713e25bb113 lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcb44 expref: -99 pid: 156957 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572076.700788:0:156957:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572076.700805:0:156957:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891500 x1875444569443072/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572076.700839:0:156957:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156957:1875444569443072:192.168.202.157@tcp:36:lfs.0 00000100:00100000:2.0:1788572076.701109:0:156957:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f6c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.795992:0:156958:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156958:1875444569443456:192.168.202.157@tcp:101:rmdir.0 00010000:00010000:3.0:1788572076.796030:0:156958:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 4/1,0 mode: --/CR res: [0x240002728:0x43:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796047:0:156958:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 4/1,0 mode: --/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796062:0:156958:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796086:0:156958:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796102:0:156958:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796111:0:156958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796120:0:156958:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.796133:0:156958:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a46527800 x1875444569443456/t0(0) o101->lustre-MDT0001-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572076.796151:0:156958:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796168:0:156958:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a43167000 type 13 mode 4 res 9663686440/67 (0 0) 00010000:00010000:3.0:1788572076.796177:0:156958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796187:0:156958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572076.796235:0:156958:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x2038:0x0], updating i_size 8192 00000080:00200000:3.0:1788572076.796240:0:156958:0:(namei.c:147:ll_iget()) got inode: [0x20001a9e3:0x2038:0x0](ffff920a50d43a88): rc = 0 00000080:00010000:3.0:1788572076.796251:0:156958:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x2038:0x0](ffff920a50d43a88) for lock 0x8364c713e25bb11a 00000080:00010000:3.0:1788572076.796272:0:156958:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x2038:0x0](ffff920a50d43a88) 00000080:00010000:3.0:1788572076.796276:0:156958:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb11a from it ffffaf7301c47e08 00010000:00010000:3.0:1788572076.796280:0:156958:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71c00/0x8364c713e25bb11a lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x2038:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcb8a expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.796289:0:156958:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71c00/0x8364c713e25bb11a lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2038:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcb8a expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572076.796332:0:156958:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 3 00000080:00200000:3.0:1788572076.796344:0:156958:0:(namei.c:2261:ll_rmdir()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00010000:00010000:3.0:1788572076.796397:0:156958:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71c00/0x8364c713e25bb11a lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2038:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcb8a expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.796472:0:156958:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71c00/0x8364c713e25bb11a lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x2038:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcb8a expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.796481:0:156958:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572076.796484:0:156958:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71c00/0x8364c713e25bb11a lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x2038:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcb8a expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572076.796516:0:156958:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572076.796524:0:156958:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3e300 x1875444569443712/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572076.796551:0:156958:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156958:1875444569443712:192.168.202.157@tcp:36:rmdir.0 00000100:00100000:3.0:1788572076.796693:0:156958:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822f180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.806719:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572076.806840:0:15192:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584181248 00000100:00100000:2.0:1788572076.806856:0:15192:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584181248 00000100:00100000:2.0:1788572076.806956:0:15192:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9471 00000100:00100000:2.0:1788572076.806969:0:15192:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584181248:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572076.807131:0:15192:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb121 lrc: 2/0,0 mode: PR/PR res: [0x200025d90:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcb98 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572076.807244:0:15192:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572076.807253:0:15192:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584181248:12345-192.168.202.157@tcp:104: Request processed in 288us (551us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572076.807262:0:15192:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9471 00000100:00100000:2.0:1788572076.823707:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572076.824057:0:33366:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788572076.824093:0:33366:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584181504 00000100:00100000:3.0:1788572076.824101:0:33366:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584181504 00000100:00100000:3.0:1788572076.824177:0:33366:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9472 00000100:00100000:3.0:1788572076.824188:0:33366:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a5188bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188053:x1875444584181504:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572076.824212:0:33366:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71e00/0x8364c713e25bb128 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x43:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcba6 expref: -99 pid: 156958 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572076.824311:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572076.824318:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a5188bc50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188053:x1875444584181504:12345-192.168.202.157@tcp:104: Request processed in 132us (628us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572076.824326:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9472 00010000:00010000:2.0:1788572076.902775:0:156961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:2.0:1788572076.902892:0:156961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.902913:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.902925:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.902937:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.902944:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.902950:0:156961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.902954:0:156961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.902960:0:156961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300dcbbe8 00010000:00010000:2.0:1788572076.902964:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.902973:0:156961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.902995:0:156961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.903015:0:156961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=16408 00000080:00200000:2.0:1788572076.903022:0:156961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.903041:0:156961:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=16392 00000080:00200000:2.0:1788572076.903045:0:156961:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572076.903059:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572076.903113:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.903122:0:156961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572076.903135:0:156961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572076.903163:0:156961:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444224:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572076.903430:0:156961:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fd80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.906318:0:156961:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444224:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572076.906350:0:156961:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906369:0:156961:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906377:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906384:0:156961:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906395:0:156961:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906401:0:156961:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47817400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572076.906406:0:156961:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906413:0:156961:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906433:0:156961:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 5/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906442:0:156961:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572076.906450:0:156961:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.906465:0:156961:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817400/0x8364c713e25bb136 lrc: 0/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.906484:0:156961:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a52892300 x1875444569444224/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572076.906503:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572076.906535:0:156961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:3.0:1788572076.906919:0:156961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572076.906940:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572076.906952:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.906967:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.906974:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572076.906981:0:156961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572076.906984:0:156961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:3.0:1788572076.906990:0:156961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300dcbb58 00010000:00010000:3.0:1788572076.906994:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572076.907112:0:156961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572076.907128:0:156961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572076.907137:0:156961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=13 00000080:00200008:3.0:1788572076.907149:0:156961:0:(file.c:6296:ll_getattr_dentry()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572076.907154:0:156961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908),name=d71a.replay-single 00000002:00010000:3.0:1788572076.907168:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x1b8b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572076.907174:0:156961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00010000:00010000:3.0:1788572076.907223:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.907232:0:156961:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.907244:0:156961:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156961 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572076.907270:0:156961:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444352:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572076.907385:0:156961:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822fc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.911449:0:156961:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444352:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572076.911501:0:156961:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.911525:0:156961:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.911535:0:156961:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.911549:0:156961:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.911562:0:156961:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.911577:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.912162:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.912173:0:156961:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572076.912192:0:156961:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3f800 x1875444569444352/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572092 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572076.912209:0:156961:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.912223:0:156961:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/7051 (0 0) 00000080:00200000:3.0:1788572076.912230:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572076.912263:0:156961:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x1b8b:0x0], updating i_size 4096 00000080:00010000:3.0:1788572076.912271:0:156961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) for lock 0x8364c713e25bb13d 00000080:00010000:3.0:1788572076.912281:0:156961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00000080:00010000:3.0:1788572076.912284:0:156961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb13d from it ffffaf7300dcbcb8 00010000:00010000:3.0:1788572076.912288:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572076.912299:0:156961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572076.914230:0:156961:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572076.914242:0:156961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.914271:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.914296:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.914314:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.914323:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.914333:0:156961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.914336:0:156961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.914344:0:156961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300dcbcb8 00010000:00010000:2.0:1788572076.914348:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.914358:0:156961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.914970:0:156961:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572076.914988:0:156961:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572076.914998:0:156961:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.915121:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.915129:0:156961:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572076.915134:0:156961:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572076.915137:0:156961:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572076.915142:0:156961:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300dcbbf8 00010000:00010000:2.0:1788572076.915145:0:156961:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.915152:0:156961:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572076.915164:0:156961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572076.915174:0:156961:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=267 00000080:00200000:2.0:1788572076.915181:0:156961:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 44 00000080:00200008:2.0:1788572076.915190:0:156961:0:(file.c:1090:ll_file_open()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00000080:00200080:2.0:1788572076.915217:0:156961:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) cmd=c03066f0 arg=7ffccaa4d500 00000080:00200000:2.0:1788572076.915233:0:156961:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) name=test stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572076.915267:0:156961:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572076.915285:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6400/0x8364c713e25aecaa lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d8 expref: -99 pid: 153007 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572076.915341:0:156961:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572076.915348:0:156961:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572076.915351:0:156961:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6800/0x8364c713e25bb13d lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcc08 expref: -99 pid: 156961 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572076.915373:0:156961:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572076.915388:0:156961:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890380 x1875444569444608/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572076.915417:0:156961:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444608:192.168.202.157@tcp:36:lfs.0 00000100:00100000:2.0:1788572076.916210:0:156961:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572076.956864:0:156961:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156961:1875444569444608:192.168.202.157@tcp:36:lfs.0 00000080:00200000:3.0:1788572076.956959:0:156961:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20001a9e3:0x2039:0x0](ffff920a50d3a1c8) 00000080:00200000:3.0:1788572076.956967:0:156961:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x2039:0x0], updating i_size 4096 00000080:00200008:3.0:1788572076.957023:0:156961:0:(file.c:479:ll_file_release()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00010000:00010000:3.0:1788572076.957064:0:156961:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a78600/0x8364c713e25aed8a lrc: 2/0,0 mode: CR/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d1 expref: -99 pid: 153019 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:3.0:1788572077.069853:0:156967:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156967:1875444569444992:192.168.202.157@tcp:101:rmdir.0 00010000:00010000:3.0:1788572077.069899:0:156967:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 4/1,0 mode: --/CR res: [0x240002728:0x44:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069916:0:156967:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 4/1,0 mode: --/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069937:0:156967:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069949:0:156967:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069966:0:156967:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069975:0:156967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.069982:0:156967:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572077.069995:0:156967:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a46524000 x1875444569444992/t0(0) o101->lustre-MDT0001-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572077.070013:0:156967:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.070028:0:156967:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a43167000 type 13 mode 4 res 9663686440/68 (0 0) 00010000:00010000:3.0:1788572077.070038:0:156967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.070047:0:156967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572077.070104:0:156967:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x2039:0x0], updating i_size 8192 00000080:00200000:3.0:1788572077.070109:0:156967:0:(namei.c:147:ll_iget()) got inode: [0x20001a9e3:0x2039:0x0](ffff920a50d3a1c8): rc = 0 00000080:00010000:3.0:1788572077.070121:0:156967:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x2039:0x0](ffff920a50d3a1c8) for lock 0x8364c713e25bb144 00000080:00010000:3.0:1788572077.070141:0:156967:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x2039:0x0](ffff920a50d3a1c8) 00000080:00010000:3.0:1788572077.070145:0:156967:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb144 from it ffffaf7300c87e08 00010000:00010000:3.0:1788572077.070149:0:156967:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71800/0x8364c713e25bb144 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x2039:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcc4e expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.070159:0:156967:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71800/0x8364c713e25bb144 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2039:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcc4e expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572077.070186:0:156967:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 3 00000080:00200000:3.0:1788572077.070194:0:156967:0:(namei.c:2261:ll_rmdir()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00010000:00010000:3.0:1788572077.070228:0:156967:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71800/0x8364c713e25bb144 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2039:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcc4e expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.070287:0:156967:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71800/0x8364c713e25bb144 lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x2039:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcc4e expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.070295:0:156967:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572077.070299:0:156967:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71800/0x8364c713e25bb144 lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x2039:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcc4e expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572077.070330:0:156967:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572077.070338:0:156967:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3f800 x1875444569445248/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572077.070365:0:156967:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156967:1875444569445248:192.168.202.157@tcp:36:rmdir.0 00000100:00100000:3.0:1788572077.070690:0:156967:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822fc00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572077.077832:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.077894:0:15193:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584182912 00000100:00100000:3.0:1788572077.077907:0:15193:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584182912 00000100:00100000:3.0:1788572077.077986:0:15193:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9473 00000100:00100000:3.0:1788572077.077999:0:15193:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584182912:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.078023:0:15193:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb14b lrc: 2/0,0 mode: PR/PR res: [0x200025d90:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcc5c expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.078129:0:15193:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.078136:0:15193:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584182912:12345-192.168.202.157@tcp:104: Request processed in 141us (319us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.078144:0:15193:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9473 00000100:00100000:3.0:1788572077.089643:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572077.091324:0:15192:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584183168 00000100:00100000:2.0:1788572077.091339:0:15192:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584183168 00000100:00100000:2.0:1788572077.091545:0:15192:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9474 00000100:00100000:2.0:1788572077.091558:0:15192:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584183168:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572077.091606:0:15192:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a71000/0x8364c713e25bb152 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x44:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcc6a expref: -99 pid: 156967 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572077.091714:0:15192:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572077.091722:0:15192:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584183168:12345-192.168.202.157@tcp:104: Request processed in 167us (2134us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572077.091730:0:15192:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9474 00000100:00100000:3.0:1788572077.117738:0:156967:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156967:1875444569445248:192.168.202.157@tcp:36:rmdir.0 00000080:00200000:3.0:1788572077.117819:0:156967:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x2039:0x0](ffff920a50d3a1c8) 00000080:00200000:2.0:1788572077.208349:0:156972:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.208394:0:156972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.208495:0:156972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.208515:0:156972:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.208525:0:156972:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.208534:0:156972:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.208537:0:156972:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.208546:0:156972:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300d17c20 00010000:00010000:2.0:1788572077.208551:0:156972:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.208561:0:156972:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.208578:0:156972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.208719:0:156972:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=24 00000080:00200000:2.0:1788572077.208727:0:156972:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.208743:0:156972:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=0 00000080:00200000:2.0:1788572077.208747:0:156972:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572077.208761:0:156972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572077.208823:0:156972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.208831:0:156972:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.208843:0:156972:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572077.208888:0:156972:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156972:1875444569446272:192.168.202.157@tcp:101:rmdir.0 00000100:00100000:2.0:1788572077.209178:0:156972:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f240 going to sleep for 11 seconds 00000100:00100000:2.0:1788572077.217950:0:156972:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156972:1875444569446272:192.168.202.157@tcp:101:rmdir.0 00010000:00010000:2.0:1788572077.217998:0:156972:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218028:0:156972:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218038:0:156972:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218064:0:156972:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218079:0:156972:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218096:0:156972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218106:0:156972:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218113:0:156972:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572077.218127:0:156972:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a52891180 x1875444569446272/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/672 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572077.218142:0:156972:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb16e lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203a:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd12 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218156:0:156972:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8250 (0 0) 00000080:00200000:2.0:1788572077.218164:0:156972:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00000002:00010000:2.0:1788572077.218251:0:156972:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200025d90:0x45:0x0]) in obj [0x200025d90:0x45:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.218318:0:156972:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817800/0x8364c713e25bb175 lrc: 3/1,0 mode: --/CR res: [0x200025d90:0x45:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218326:0:156972:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817800/0x8364c713e25bb175 lrc: 3/1,0 mode: --/CR res: [0x200025d90:0x45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.218335:0:156972:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817800/0x8364c713e25bb175 lrc: 3/1,0 mode: --/CR res: [0x200025d90:0x45:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 156972 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572077.218360:0:156972:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156972:1875444569446400:192.168.202.157@tcp:101:rmdir.0 00000100:00100000:2.0:1788572077.218476:0:156972:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f240 going to sleep for 11 seconds 00000100:00100000:3.0:1788572077.265041:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.265098:0:33366:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584184576 00000100:00100000:3.0:1788572077.265111:0:33366:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584184576 00000100:00100000:3.0:1788572077.265176:0:33366:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9475 00000100:00100000:3.0:1788572077.265188:0:33366:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188048:x1875444584184576:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.265212:0:33366:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817800/0x8364c713e25bb175 lrc: 2/0,0 mode: PR/PR res: [0x200025d90:0x45:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcd20 expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.266209:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.266227:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3e680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188048:x1875444584184576:12345-192.168.202.157@tcp:104: Request processed in 1037us (1197us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.266238:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9475 00000100:00100000:3.0:1788572077.276177:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.276220:0:15193:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584184832 00000100:00100000:3.0:1788572077.276232:0:15193:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584184832 00000100:00100000:3.0:1788572077.276299:0:15193:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9476 00000100:00100000:3.0:1788572077.276314:0:15193:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584184832:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.276342:0:15193:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb17c lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x45:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcd2e expref: -99 pid: 156972 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.276437:0:15193:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.276445:0:15193:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584184832:12345-192.168.202.157@tcp:104: Request processed in 135us (286us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.276453:0:15193:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9476 00000080:00200000:3.0:1788572077.368313:0:156978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572077.368357:0:156978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572077.368385:0:156978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572077.368405:0:156978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572077.368415:0:156978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572077.368423:0:156978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572077.368427:0:156978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:3.0:1788572077.368436:0:156978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300c87b58 00010000:00010000:3.0:1788572077.368441:0:156978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:3.0:1788572077.368451:0:156978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:3.0:1788572077.368772:0:156978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572077.368783:0:156978:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=29 00000080:00200000:3.0:1788572077.368791:0:156978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:3.0:1788572077.368803:0:156978:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=13 00000080:00200000:3.0:1788572077.368808:0:156978:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:3.0:1788572077.368824:0:156978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572077.368897:0:156978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb183 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.368907:0:156978:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb183 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.368918:0:156978:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6c00/0x8364c713e25bb183 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156978 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572077.368958:0:156978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156978:1875444569447168:192.168.202.157@tcp:101:lfs.0 00000080:00200000:2.0:1788572077.431330:0:156978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x1b8b:0x0], updating i_size 4096 00000080:00010000:2.0:1788572077.431340:0:156978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) for lock 0x8364c713e25bb191 00000080:00010000:2.0:1788572077.431349:0:156978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00000080:00010000:2.0:1788572077.431352:0:156978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb191 from it ffffaf7300c87cb8 00010000:00010000:2.0:1788572077.431356:0:156978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb191 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcd90 expref: -99 pid: 156978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.431366:0:156978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb191 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcd90 expref: -99 pid: 156978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572077.433311:0:156978:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572077.433323:0:156978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.433350:0:156978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.433372:0:156978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.433388:0:156978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.433398:0:156978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.433405:0:156978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.433524:0:156978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.433535:0:156978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300c87cb8 00010000:00010000:2.0:1788572077.433539:0:156978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.433556:0:156978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.433999:0:156978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.434013:0:156978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.434023:0:156978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.434033:0:156978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.434051:0:156978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.434056:0:156978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.434058:0:156978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.434063:0:156978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300c87bf8 00010000:00010000:2.0:1788572077.434066:0:156978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.434074:0:156978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.434085:0:156978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.434094:0:156978:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=267 00000080:00200000:2.0:1788572077.434102:0:156978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 44 00000080:00200008:2.0:1788572077.434110:0:156978:0:(file.c:1090:ll_file_open()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00000080:00200080:2.0:1788572077.434133:0:156978:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) cmd=c03066f0 arg=7ffd8be39860 00000080:00200000:2.0:1788572077.434148:0:156978:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) name=test stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572077.434179:0:156978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb191 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcd90 expref: -99 pid: 156978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572077.434195:0:156978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6400/0x8364c713e25aecaa lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d8 expref: -99 pid: 153007 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.434317:0:156978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb191 lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcd90 expref: -99 pid: 156978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572077.434326:0:156978:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572077.434331:0:156978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70c00/0x8364c713e25bb191 lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcd90 expref: -99 pid: 156978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572077.434352:0:156978:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572077.434367:0:156978:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52892d80 x1875444569447680/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572077.434398:0:156978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156978:1875444569447680:192.168.202.157@tcp:36:lfs.0 00000100:00100000:2.0:1788572077.434538:0:156978:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572077.459898:0:156978:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156978:1875444569447680:192.168.202.157@tcp:36:lfs.0 00000080:00200000:2.0:1788572077.459987:0:156978:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20001a9e3:0x203b:0x0](ffff920a50d2ec08) 00000080:00200000:2.0:1788572077.459995:0:156978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x203b:0x0], updating i_size 4096 00000080:00200008:2.0:1788572077.460319:0:156978:0:(file.c:479:ll_file_release()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00010000:00010000:2.0:1788572077.460365:0:156978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a78600/0x8364c713e25aed8a lrc: 2/0,0 mode: CR/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d1 expref: -99 pid: 153019 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.478997:0:156982:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.479541:0:156982:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.479569:0:156982:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.479765:0:156982:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.479777:0:156982:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.479787:0:156982:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.479791:0:156982:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.479800:0:156982:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300d17c20 00010000:00010000:2.0:1788572077.479805:0:156982:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.479816:0:156982:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.479835:0:156982:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.479845:0:156982:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=24 00000080:00200000:2.0:1788572077.479862:0:156982:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.479880:0:156982:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=0 00000080:00200000:2.0:1788572077.479884:0:156982:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572077.479900:0:156982:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572077.479957:0:156982:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817600/0x8364c713e25bb198 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156982 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.479966:0:156982:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817600/0x8364c713e25bb198 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156982 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.479983:0:156982:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817600/0x8364c713e25bb198 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156982 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572077.480018:0:156982:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156982:1875444569447808:192.168.202.157@tcp:101:rmdir.0 00000100:00100000:2.0:1788572077.481692:0:156982:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572077.529168:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.529307:0:15193:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584186240 00000100:00100000:3.0:1788572077.529533:0:15193:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584186240 00000100:00100000:3.0:1788572077.529862:0:15193:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9477 00000100:00100000:3.0:1788572077.529875:0:15193:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a52892a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188048:x1875444584186240:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.529899:0:15193:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79200/0x8364c713e25bb19f lrc: 2/0,0 mode: PR/PR res: [0x200025d90:0x46:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcde4 expref: -99 pid: 156982 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.530005:0:15193:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.530012:0:15193:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a52892a00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188048:x1875444584186240:12345-192.168.202.157@tcp:104: Request processed in 141us (863us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.530020:0:15193:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9477 00000100:00100000:3.0:1788572077.544059:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.544124:0:15192:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584186496 00000100:00100000:3.0:1788572077.544137:0:15192:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584186496 00000100:00100000:3.0:1788572077.544206:0:15192:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9478 00000100:00100000:3.0:1788572077.544217:0:15192:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188053:x1875444584186496:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.544241:0:15192:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1a6 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x46:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcdf2 expref: -99 pid: 156982 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.544454:0:15192:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.544463:0:15192:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3ca80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188053:x1875444584186496:12345-192.168.202.157@tcp:104: Request processed in 248us (1011us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.544481:0:15192:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9478 00000100:00100000:2.0:1788572077.665429:0:156991:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569448704:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572077.666129:0:156991:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666152:0:156991:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666161:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666169:0:156991:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666182:0:156991:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666189:0:156991:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a51a70600 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:2.0:1788572077.666194:0:156991:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666201:0:156991:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666214:0:156991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 5/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666221:0:156991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572077.666231:0:156991:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.666238:0:156991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb1ad lrc: 0/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572077.666263:0:156991:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a46525180 x1875444569448704/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572077.666283:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572077.666317:0:156991:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:2.0:1788572077.666408:0:156991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.666427:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.666438:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.666451:0:156991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.666459:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.667100:0:156991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.667104:0:156991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.667111:0:156991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300da3be8 00010000:00010000:2.0:1788572077.667116:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.667128:0:156991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.667145:0:156991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.667155:0:156991:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=16408 00000080:00200000:2.0:1788572077.667164:0:156991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.667186:0:156991:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=16392 00000080:00200000:2.0:1788572077.667191:0:156991:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572077.667206:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572077.667257:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.667265:0:156991:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.667275:0:156991:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572077.667305:0:156991:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569448832:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572077.667419:0:156991:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572077.670706:0:156991:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569448832:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572077.670735:0:156991:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670750:0:156991:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670763:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670771:0:156991:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x10206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670781:0:156991:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670787:0:156991:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816200 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:2.0:1788572077.670793:0:156991:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670799:0:156991:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 4/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x206400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670807:0:156991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 5/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x20e400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670815:0:156991:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572077.670822:0:156991:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 3/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.670828:0:156991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816200/0x8364c713e25bb1b4 lrc: 0/0,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x2 rrc: 4 type: IBT flags: 0x4a0f400000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572077.670860:0:156991:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a52891c00 x1875444569448832/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/584 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572077.670876:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x1b8b:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1788572077.670911:0:156991:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00000080:00200000:2.0:1788572077.671026:0:156991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.671053:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.671063:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.671075:0:156991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.671082:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.671087:0:156991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.671090:0:156991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.671095:0:156991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300da3b58 00010000:00010000:2.0:1788572077.671099:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.671107:0:156991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.671121:0:156991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.671129:0:156991:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=13 00000080:00200008:2.0:1788572077.671140:0:156991:0:(file.c:6296:ll_getattr_dentry()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572077.671145:0:156991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908),name=d71a.replay-single 00000002:00010000:2.0:1788572077.671157:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x1b8b:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.671162:0:156991:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/7051 (0 0) 00010000:00010000:2.0:1788572077.671207:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.671221:0:156991:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.671233:0:156991:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 156991 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572077.671259:0:156991:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569448960:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572077.671359:0:156991:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572077.679542:0:156991:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569448960:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572077.679575:0:156991:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679626:0:156991:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679636:0:156991:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679650:0:156991:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679663:0:156991:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679678:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679691:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679698:0:156991:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572077.679711:0:156991:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a52891c00 x1875444569448960/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572077.679728:0:156991:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679738:0:156991:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/7051 (0 0) 00000080:00200000:2.0:1788572077.679745:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572077.679778:0:156991:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x1b8b:0x0], updating i_size 4096 00000080:00010000:2.0:1788572077.679785:0:156991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) for lock 0x8364c713e25bb1bb 00000080:00010000:2.0:1788572077.679794:0:156991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00000080:00010000:2.0:1788572077.679798:0:156991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1bb from it ffffaf7300da3cb8 00010000:00010000:2.0:1788572077.679801:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.679811:0:156991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572077.680632:0:156991:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572077.680642:0:156991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.680667:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.680685:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.680699:0:156991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.680706:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.680711:0:156991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.680715:0:156991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.680720:0:156991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300da3cb8 00010000:00010000:2.0:1788572077.680723:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.680731:0:156991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.680917:0:156991:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.680925:0:156991:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.680932:0:156991:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.680940:0:156991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.680948:0:156991:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.680952:0:156991:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.680955:0:156991:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.680958:0:156991:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300da3bf8 00010000:00010000:2.0:1788572077.680962:0:156991:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.680968:0:156991:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.680977:0:156991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.680984:0:156991:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=267 00000080:00200000:2.0:1788572077.680990:0:156991:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 44 00000080:00200008:2.0:1788572077.680999:0:156991:0:(file.c:1090:ll_file_open()) START file d71a.replay-single:[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags 300000 00000080:00200080:2.0:1788572077.681022:0:156991:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) cmd=c03066f0 arg=7ffc35870de0 00000080:00200000:2.0:1788572077.681040:0:156991:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) name=test stripe_offset=0 stripe_count=2, hash_type=0 00010000:00010000:2.0:1788572077.681071:0:156991:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572077.681086:0:156991:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6400/0x8364c713e25aecaa lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x11/0x0 rrc: 5 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98d8 expref: -99 pid: 153007 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.681135:0:156991:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572077.681142:0:156991:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572077.681145:0:156991:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1bb lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bce54 expref: -99 pid: 156991 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572077.681165:0:156991:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572077.681177:0:156991:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891c00 x1875444569449216/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 552/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572077.681210:0:156991:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:156991:1875444569449216:192.168.202.157@tcp:36:lfs.0 00000100:00100000:2.0:1788572077.681320:0:156991:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f300 going to sleep for 11 seconds 00000080:00200000:2.0:1788572077.703513:0:157001:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572077.703556:0:157001:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572077.703618:0:157001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.703642:0:157001:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.703653:0:157001:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572077.703661:0:157001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572077.703665:0:157001:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25b8de9 00000080:00010000:2.0:1788572077.703675:0:157001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25b8de9 from it ffffaf7300d4bc20 00010000:00010000:2.0:1788572077.703679:0:157001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1788572077.703690:0:157001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79c00/0x8364c713e25b8de9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109b98df expref: -99 pid: 155108 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1788572077.703706:0:157001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.703715:0:157001:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d71a.replay-single, flags=24 00000080:00200000:2.0:1788572077.703721:0:157001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 1 00000080:00200000:2.0:1788572077.703733:0:157001:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), flags=0 00000080:00200000:2.0:1788572077.703738:0:157001:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), intent=getattr 00000002:00010000:2.0:1788572077.703752:0:157001:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: test,[0x0:0x0:0x0]) in obj [0x20001a9e3:0x1b8b:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572077.703814:0:157001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 157001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.703822:0:157001:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 157001 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572077.703834:0:157001:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 157001 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572077.705227:0:157001:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157001:1875444569449344:192.168.202.157@tcp:101:rmdir.0 00000100:00100000:3.0:1788572077.705483:0:157001:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822f000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572077.755719:0:157001:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157001:1875444569449728:192.168.202.157@tcp:101:rmdir.0 00010000:00010000:3.0:1788572077.755752:0:157001:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 4/1,0 mode: --/CR res: [0x240002728:0x47:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755766:0:157001:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 4/1,0 mode: --/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755780:0:157001:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755793:0:157001:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755807:0:157001:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755815:0:157001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 5/2,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755822:0:157001:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 4/1,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572077.755835:0:157001:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a528cea00 x1875444569449728/t0(0) o101->lustre-MDT0001-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572093 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rmdir.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572077.755864:0:157001:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755885:0:157001:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a43167000 type 13 mode 4 res 9663686440/71 (0 0) 00010000:00010000:3.0:1788572077.755894:0:157001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 3/1,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.755905:0:157001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572077.755957:0:157001:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x203c:0x0], updating i_size 8192 00000080:00200000:3.0:1788572077.755963:0:157001:0:(namei.c:147:ll_iget()) got inode: [0x20001a9e3:0x203c:0x0](ffff920a50d45b88): rc = 0 00000080:00010000:3.0:1788572077.756721:0:157001:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203c:0x0](ffff920a50d45b88) for lock 0x8364c713e25bb1c2 00000080:00010000:3.0:1788572077.756745:0:157001:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203c:0x0](ffff920a50d45b88) 00000080:00010000:3.0:1788572077.756749:0:157001:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1c2 from it ffffaf7300d4be08 00010000:00010000:3.0:1788572077.756754:0:157001:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bce9a expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572077.756766:0:157001:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bce9a expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572077.756799:0:157001:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908), inode mode 41ed mask 3 00000080:00200000:3.0:1788572077.756819:0:157001:0:(namei.c:2261:ll_rmdir()) VFS Op:name=test, dir=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00010000:00010000:3.0:1788572077.756871:0:157001:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203c:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bce9a expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.757787:0:157001:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x203c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bce9a expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.757803:0:157001:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572077.757807:0:157001:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817e00/0x8364c713e25bb1c2 lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x203c:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bce9a expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572077.757848:0:157001:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572077.757864:0:157001:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3c380 x1875444569450368/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572077.757894:0:157001:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job rmdir:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157001:1875444569450368:192.168.202.157@tcp:36:rmdir.0 00000100:00100000:3.0:1788572077.758540:0:157001:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4822f000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572077.765387:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.765491:0:15193:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:3.0:1788572077.765511:0:15193:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584187904 00000100:00100000:3.0:1788572077.765522:0:15193:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584187904 00000100:00100000:3.0:1788572077.765578:0:15193:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9479 00000100:00100000:3.0:1788572077.765616:0:15193:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a47ae0c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584187904:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.765640:0:15193:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a78000/0x8364c713e25bb1c9 lrc: 2/0,0 mode: PR/PR res: [0x200025d90:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bcea8 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.765723:0:15193:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.765729:0:15193:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a47ae0c50 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:188833:x1875444584187904:12345-192.168.202.157@tcp:104: Request processed in 143us (357us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.765737:0:15193:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9479 00000100:00100000:3.0:1788572077.776851:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572077.776901:0:33366:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584188160 00000100:00100000:3.0:1788572077.776913:0:33366:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584188160 00000100:00100000:3.0:1788572077.777156:0:33366:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9480 00000100:00100000:3.0:1788572077.777168:0:33366:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584188160:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572077.777193:0:33366:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.777288:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572077.777296:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3df80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d+5:189111:x1875444584188160:12345-192.168.202.157@tcp:104: Request processed in 130us (470us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572077.777304:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9480 00010000:00010000:3.0:1788572077.777899:0:15200:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572077.777909:0:15200:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.777921:0:15200:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572077.777935:0:15200:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.777946:0:15200:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.777956:0:15200:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 3/0,0 mode: PR/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.778151:0:15200:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 2/0,0 mode: --/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.778159:0:15200:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572077.778186:0:15200:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 1/0,0 mode: --/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.778193:0:15200:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a42f86800 lock: ffff920a60a78800/0x8364c713e25bb1d0 lrc: 0/0,0 mode: --/PR res: [0x240002728:0x47:0x0].0x0 bits 0x12/0x12 rrc: 2 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bceb6 expref: -99 pid: 157001 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572077.778217:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572077.778218:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572077.778220:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572077.778235:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569450624:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572077.784880:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569450624:192.168.202.157@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572078.747846:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572079.583607:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569450752:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.583724:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451008:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.583760:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451264:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.583803:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572079.583809:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569450880:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.583831:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451136:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.592072:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569450752:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.607958:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569450880:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.607981:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451136:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.607992:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451008:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572079.608003:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451264:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572081.823650:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572084.700916:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451520:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.701006:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.701033:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788572084.701038:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569451392:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.701065:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451648:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.701086:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451904:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.705275:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569451392:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.705298:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451648:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.705309:0:2399:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 330712481930 gen 11 00000100:00100000:3.0:1788572084.705399:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451520:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572084.705411:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572084.708109:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569451904:192.168.202.157@tcp:400:kworker.0 00000001:02000400:2.0:1788572085.521628:0:157580:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 21:34:44 (1788572084) 00010000:00010000:3.0:1788572085.915870:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572086.327490:0:157593:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524a80 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569452160:192.168.202.157@tcp:101:multiop.0 00000100:00100000:2.0:1788572086.327966:0:157593:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896a80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572086.343882:0:157593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524a80 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569452160:192.168.202.157@tcp:101:multiop.0 00010000:00010000:2.0:1788572086.343912:0:157593:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343947:0:157593:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/0,1 mode: --/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343956:0:157593:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/0,1 mode: --/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343967:0:157593:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/0,1 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343977:0:157593:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/0,1 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343991:0:157593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 5/1,1 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.343999:0:157593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 5/1,1 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.344007:0:157593:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572086.344019:0:157593:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a46524a80 x1875444569452160/t330712481931(330712481931) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 664/664 e 0 to 0 dl 1788572102 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572086.344030:0:157593:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46524a80 x1875444569452160/t330712481931(330712481931) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 664/664 e 0 to 0 dl 1788572102 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572086.344043:0:157593:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572086.344056:0:157593:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.344067:0:157593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8253 (0 0) 00000080:00200000:2.0:1788572086.344074:0:157593:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572086.344090:0:157593:0:(namei.c:1628:ll_create_it()) VFS Op:name=f73a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), intent=open|creat 00000080:00200000:2.0:1788572086.344122:0:157593:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) 00000080:00200000:2.0:1788572086.344204:0:157593:0:(file.c:6843:ll_layout_conf()) [0x20001a9e3:0x203d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572086.344210:0:157593:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) 00000080:00010000:2.0:1788572086.344213:0:157593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) for lock 0x8364c713e25bb1d7 00000080:00010000:2.0:1788572086.344224:0:157593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) 00000080:00200008:2.0:1788572086.344240:0:157593:0:(file.c:1090:ll_file_open()) START file f73a.replay-single:[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88), flags 100102 00000002:00100000:2.0:1788572086.344261:0:157593:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46524a80 x1875444569452160/t330712481931(330712481931) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572102 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572086.344303:0:157593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1d7 from it ffff920a4821b720 00010000:00010000:2.0:1788572086.344308:0:157593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572086.344317:0:157593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a71400/0x8364c713e25bb1d7 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bcefc expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572086.344340:0:157593:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572087.521100:0:157604:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572087.521121:0:157604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.521152:0:157604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.521233:0:157604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 157604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.521244:0:157604:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 157604 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.521256:0:157604:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 157604 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572087.521290:0:157604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157604:1875444569452800:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572087.521410:0:157604:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572087.531087:0:157604:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 330712481932 gen 11 00000100:00100000:2.0:1788572087.531130:0:157604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157604:1875444569452800:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572087.531158:0:157604:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531173:0:157604:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531187:0:157604:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531199:0:157604:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531215:0:157604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531223:0:157604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531230:0:157604:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572087.531245:0:157604:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a52891180 x1875444569452800/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572103 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572087.531263:0:157604:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531281:0:157604:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572087.531290:0:157604:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572087.531368:0:157604:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1788572087.531376:0:157604:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.531382:0:157604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.531386:0:157604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf730165fcb8 00010000:00010000:2.0:1788572087.531390:0:157604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572087.531400:0:157604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572087.534226:0:157604:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572087.534240:0:157604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.534271:0:157604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.534294:0:157604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572087.534318:0:157604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572087.534327:0:157604:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572087.534337:0:157604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.534341:0:157604:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.534350:0:157604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf730165fcb8 00010000:00010000:2.0:1788572087.534354:0:157604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572087.534366:0:157604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572087.535759:0:157604:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.535789:0:157604:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.535806:0:157604:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572087.535821:0:157604:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572087.535828:0:157604:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572087.535834:0:157604:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.535837:0:157604:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.535844:0:157604:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf730165fbb0 00010000:00010000:2.0:1788572087.535847:0:157604:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572087.535856:0:157604:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572087.535870:0:157604:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572087.535881:0:157604:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), flags 100000 00000080:00200080:2.0:1788572087.536016:0:157604:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffccd9d5510 00800000:00000080:2.0:1788572087.536037:0:157604:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4c00 uarg=0000000000000000 00800000:00000004:2.0:1788572087.536046:0:157604:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: age 8836, max_age 8845 00000100:00100000:2.0:1788572087.536099:0:157604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157604:1875444569452928:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572087.536213:0:157604:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572087.539817:0:157604:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157604:1875444569452928:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572087.539857:0:157604:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: update ffff920a54025198 cache blocks 317072/353529 objects 1023637/1024000 00000080:00200080:2.0:1788572087.539949:0:157604:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffccd9d5510 00800000:00000080:2.0:1788572087.539970:0:157604:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4c00 uarg=0000000000000000 00800000:00000004:2.0:1788572087.539979:0:157604:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a42f86800: age 8836, max_age 8845 00000100:00100000:2.0:1788572087.541977:0:157604:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157604:1875444569453056:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572087.542183:0:157604:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896b40 going to sleep for 11 seconds 00000080:00200000:2.0:1788572087.628336:0:157611:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.628375:0:157611:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.628400:0:157611:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572087.628421:0:157611:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572087.628431:0:157611:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572087.628439:0:157611:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.628443:0:157611:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.628468:0:157611:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf7301d77bd0 00010000:00010000:2.0:1788572087.628473:0:157611:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572087.628484:0:157611:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572087.628506:0:157611:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572087.628522:0:157611:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=1536 00000080:00200000:2.0:1788572087.628526:0:157611:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.628532:0:157611:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.628539:0:157611:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572087.628548:0:157611:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:2.0:1788572087.628555:0:157611:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572087.628559:0:157611:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.628561:0:157611:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.628565:0:157611:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf7301d77cd0 00010000:00010000:2.0:1788572087.628568:0:157611:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:2.0:1788572087.628576:0:157611:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572087.628583:0:157611:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572087.628620:0:157611:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572087.628626:0:157611:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572087.628632:0:157611:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572087.628647:0:157611:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572087.628653:0:157611:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572087.628657:0:157611:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572087.628660:0:157611:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb1ec 00000080:00010000:2.0:1788572087.628663:0:157611:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb1ec from it ffffaf7301d77db0 00010000:00010000:2.0:1788572087.628666:0:157611:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572087.628673:0:157611:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572087.628681:0:157611:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572087.628686:0:157611:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=33188, excl=1 00000080:00200000:2.0:1788572087.628690:0:157611:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8) mode 100644 dev 0 00010000:00010000:2.0:1788572087.628722:0:157611:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572087.628785:0:157611:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572087.628793:0:157611:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572087.628796:0:157611:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47817c00/0x8364c713e25bb1ec lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bcf3b expref: -99 pid: 157604 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:2.0:1788572087.628826:0:157611:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572087.628843:0:157611:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890700 x1875444569453568/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572087.628873:0:157611:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job mcreate:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157611:1875444569453568:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:2.0:1788572087.628981:0:157611:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896000 going to sleep for 11 seconds 00000100:00100000:2.0:1788572087.637807:0:157611:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job mcreate:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157611:1875444569453568:192.168.202.157@tcp:36:mcreate.0 00000080:00200000:2.0:1788572087.637879:0:157611:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20001a9e3:0x203e:0x0](ffff920a50d29148) 00000080:00200000:2.0:1788572087.637932:0:157611:0:(namei.c:2070:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, unhashed 1 00010000:00010000:3.0:1788572090.025061:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572093.085038:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572094.110197:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572094.940469:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569454848:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572094.940481:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569454976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572094.940579:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569455232:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572094.940605:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455104:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572094.940632:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572094.946480:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569454976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572094.946480:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455104:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572094.946511:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572094.946526:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569454848:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572094.946870:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569455232:192.168.202.157@tcp:400:kworker.0 00000001:02000400:2.0:1788572096.370033:0:157687:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:2.0:1788572097.183247:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572098.216301:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572100.061176:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455616:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572100.061276:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455872:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572100.061325:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788572100.061333:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569455488:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572100.061581:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572100.061646:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569456000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.072694:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455616:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.072716:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455872:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.072747:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569455488:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.072759:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569455744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572100.072770:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569456000:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572101.276865:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572102.300148:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572105.186873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569456768:192.168.202.157@tcp:38:ptlrpcd_00_01.0 00000100:00080000:2.0:1788572105.206455:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572105.206471:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572105.206477:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569456768:192.168.202.157@tcp:38:ptlrpcd_00_01.0 00010000:00010000:2.0:1788572105.371836:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572106.396108:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572109.468606:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572110.301203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572110.301256:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457408:192.168.202.157@tcp:9:kworker.0 00000100:00080000:2.0:1788572110.308462:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572110.308476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572110.308481:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457024:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572110.313829:0:2396:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff920a528ce680 x1875444569457408/t0(0) o9->lustre-OST0001-osc-ffff920a42f86800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572120 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572110.313856:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a45969800 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572110.313862:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457408:192.168.202.157@tcp:9:kworker.0 00010000:00010000:3.0:1788572110.491950:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572113.564407:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572114.588003:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572115.422690:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572115.432542:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572115.432563:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572115.432570:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569457664:192.168.202.157@tcp:38:kworker.0 00010000:00010000:2.0:1788572116.638543:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572117.664830:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572118.683851:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572120.541401:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569458176:192.168.202.157@tcp:38:kworker.0 00010000:00010000:2.0:1788572120.732574:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572121.614754:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 712ceec8-7c82-4602-bd77-3db1811a6b63->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572121.614760:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572121.614763:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572121.614768:0:124398:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 8797 00000100:00080000:3.0:1788572121.614782:0:124398:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a4596a800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572121.614813:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3f800 x1875444569458560/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572121.614836:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572121.614839:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572121.614841:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572121.614845:0:124398:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8879 00000100:00080000:3.0:1788572121.614849:0:124398:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572121.614853:0:124398:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572121.614862:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572121.614871:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3e300 x1875444569458688/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572121.615865:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569458816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572121.621110:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572121.621134:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569458944:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572121.631968:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569458944:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572121.631984:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569458816:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572121.755827:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572122.780148:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572124.827878:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572125.852375:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572126.684258:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 712ceec8-7c82-4602-bd77-3db1811a6b63->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572126.684269:0:103578:0:(pinger.c:247:ptlrpc_pinger_process_import()) 712ceec8-7c82-4602-bd77-3db1811a6b63->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572126.684273:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572126.684277:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572126.684280:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572126.684290:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8880 00000100:00080000:2.0:1788572126.684299:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572126.684312:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572126.684324:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572126.684360:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569459072/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572126.684484:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569459200:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572126.684566:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572126.684572:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569459328:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572126.697287:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569459328:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572126.697302:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569459200:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572126.882462:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572128.923891:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572129.949897:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572130.971967:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572131.814984:0:33368:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572131.815005:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572131.815007:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572131.815010:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572131.815379:0:157963:0:(import.c:1642:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.157@tcp to MGS@192.168.202.157@tcp 00000100:00100000:2.0:1788572131.815384:0:157963:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572131.815386:0:157963:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788572131.815400:0:157963:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25ba67e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109b6371 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572131.815410:0:157963:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25ba67e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109b6371 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572131.815417:0:157963:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572131.815435:0:157963:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572131.815443:0:157963:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25ba67e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109b6371 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572131.815455:0:157963:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a78a00/0x8364c713e25ba68c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109b63c5 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572131.815462:0:157963:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a78a00/0x8364c713e25ba68c lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109b63c5 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572131.815467:0:157963:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572131.815470:0:157963:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572131.815473:0:157963:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a78a00/0x8364c713e25ba68c lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109b63c5 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572131.815486:0:157963:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a47816000/0x8364c713e25ba693 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109b6577 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572131.815491:0:157963:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a47816000/0x8364c713e25ba693 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109b6577 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572131.815496:0:157963:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572131.815499:0:157963:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572131.815502:0:157963:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a47816000/0x8364c713e25ba693 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109b6577 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572131.815512:0:157963:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25ba685 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109b6394 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572131.815522:0:157963:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25ba685 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109b6394 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572131.815526:0:157963:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788572131.815529:0:157963:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572131.815532:0:157963:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25ba685 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109b6394 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572131.815544:0:157963:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788572131.815547:0:157963:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788572131.815549:0:157963:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572131.815552:0:157963:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788572131.815555:0:157963:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) MGC192.168.202.157@tcp: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00010000:00010000:2.0:1788572131.996050:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572133.024360:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572134.046356:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572135.072495:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572136.091815:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572137.116652:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572139.167100:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572140.188856:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572141.216888:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572142.046091:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572142.046100:0:124398:0:(pinger.c:247:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788572142.046460:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569461760:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572142.046635:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569462016:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572142.046676:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572142.046683:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569461888:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572142.050484:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569461760:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572142.052019:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569461888:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572142.055257:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569462016:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572143.260616:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572144.284107:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572145.309676:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572147.334477:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569454080:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572147.335415:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569462656:192.168.202.157@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788572147.335508:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46525c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569462784:192.168.202.157@tcp:101:ldlm_lock_repla.0 00000100:02000000:2.0:1788572147.354965:0:2396:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff920a42f86800: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00080000:2.0:1788572147.354975:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000008f45afdf): req@ffff920a528cdf80 x1875444569461632/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572147.355002:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569462912:192.168.202.157@tcp:400:ptlrpcd_rcv.0 00010000:00010000:3.0:1788572147.356580:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572147.386020:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569463424:192.168.202.157@tcp:8:lfs.0 00010000:00010000:2.0:1788572148.381640:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572149.406260:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572151.451958:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572152.284710:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569463808:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.284829:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464064:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.284868:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788572152.284874:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569463680:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.284906:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569463936:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.284929:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464192:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.295273:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569463680:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.295307:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569463936:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.295319:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464192:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.295333:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569463808:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572152.295345:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464064:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572152.475994:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572153.502858:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572154.527914:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572155.551879:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572156.015230:0:158304:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788572156.585427:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572157.404549:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569464320:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572157.404666:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464576:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572157.404697:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464832:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572157.404730:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572157.404737:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464448:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572157.404761:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464704:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572157.411069:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464448:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572157.411086:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464704:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572157.411095:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569464320:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572157.411104:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464576:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572157.411114:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569464832:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572157.595901:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572158.152571:0:158452:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788572158.619960:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572159.643901:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572160.685715:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200080:3.0:1788572160.955925:0:158491:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd07476770 00800000:00000080:3.0:1788572160.955950:0:158491:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bf800 uarg=0000000000000000 00800000:00000004:3.0:1788572160.955961:0:158491:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: age 8905, max_age 8918 00000100:00100000:3.0:1788572160.956034:0:158491:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:158491:1875444569464960:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572160.956201:0:158491:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51a5e000 going to sleep for 11 seconds 00000080:00200008:3.0:1788572161.032328:0:157593:0:(llite_lib.c:2427:ll_setattr_raw()) START file f73a.replay-single:[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) current size 0, valid attrs 41, mode 8000, uid 1, gid 0, new size 18446744071817554661, atime 1788572161.000000000, mtime 1788572161.000000000, ctime 1788572161.000000000 00000020:00100000:3.0:1788572161.032432:0:157593:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572161.032451:0:157593:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3fb80 x1875444569465600/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572161.032486:0:157593:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569465600:192.168.202.157@tcp:36:multiop.0 00000100:00100000:3.0:1788572161.032713:0:157593:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51a5e000 going to sleep for 11 seconds 00000080:00200000:3.0:1788572161.086246:0:157593:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x203d:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788572161.086268:0:157593:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f73a.replay-single:[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) current size 0, valid attrs 10000041, mode 8000, uid 1, gid 0, new size 18446744071817554661, atime 1788572161.000000000, mtime 1788572161.000000000, ctime 1788572161.000000000, rc 0 00000080:00200008:3.0:1788572161.086311:0:157593:0:(file.c:6296:ll_getattr_dentry()) START file f73a.replay-single:[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572161.086319:0:157593:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88),name=f73a.replay-single 00000002:00010000:3.0:1788572161.086345:0:157593:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x203d:0x0]) in obj [0x20001a9e3:0x203d:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572161.086364:0:157593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/8253 (0 0) 00010000:00010000:3.0:1788572161.086444:0:157593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 157593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.086455:0:157593:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 157593 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.086466:0:157593:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 157593 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572161.086491:0:157593:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569465984:192.168.202.157@tcp:101:multiop.0 00000100:00100000:3.0:1788572161.086576:0:157593:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51a5e0c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572161.104368:0:157593:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569465984:192.168.202.157@tcp:101:multiop.0 00010000:00010000:3.0:1788572161.104409:0:157593:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104424:0:157593:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104439:0:157593:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104451:0:157593:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104468:0:157593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104478:0:157593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104486:0:157593:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572161.104500:0:157593:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3ea00 x1875444569465984/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572177 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572161.104515:0:157593:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572161.104530:0:157593:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104549:0:157593:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8253 (0 0) 00000080:00200000:3.0:1788572161.104556:0:157593:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x203d:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572161.104652:0:157593:0:(file.c:6843:ll_layout_conf()) [0x20001a9e3:0x203d:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572161.104659:0:157593:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) for lock 0x8364c713e25bb22b 00000080:00010000:3.0:1788572161.104666:0:157593:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203d:0x0](ffff920a50d2ba88) 00000080:00010000:3.0:1788572161.104670:0:157593:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb22b from it ffffaf7300dd3cd8 00010000:00010000:3.0:1788572161.104675:0:157593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572161.104684:0:157593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7000/0x8364c713e25bb22b lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203d:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd404 expref: -99 pid: 157593 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572161.104726:0:157593:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x203d:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572161.104738:0:157593:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20001a9e3:0x203d:0x0] 00000008:00010000:3.0:1788572161.104764:0:157593:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572161.104768:0:157593:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff920a51a38ab0, osc lock ffff920a45802390, flags 0x400001002 00010000:00010000:3.0:1788572161.104790:0:157593:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 0000000086415aae/0x8364c713e25bb1e5 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x19c2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bcf18 expref: -99 pid: 157595 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572161.104808:0:157593:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 0000000086415aae/0x8364c713e25bb1e5 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x19c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bcf18 expref: -99 pid: 157595 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788572161.104823:0:157593:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572161.104828:0:157593:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 0000000086415aae/0x8364c713e25bb1e5 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x19c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bcf18 expref: -99 pid: 157595 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572161.104842:0:157593:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff920a42f86800 lock: 0000000086415aae/0x8364c713e25bb1e5 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x19c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bcf18 expref: -99 pid: 157595 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788572161.104854:0:157593:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572161.104867:0:157593:0:(file.c:1775:ll_merge_attr_nolock()) [0x20001a9e3:0x203d:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788572161.104871:0:157593:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff920a540c72e0 (ffff920a459526c8/0) at cl_lock_release():245 00000020:00010000:3.0:1788572161.104875:0:157593:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a540c72e0 (ffff920a459526c8/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572161.104879:0:157593:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a51a38ab0 (ffff920a529b42d8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572161.104882:0:157593:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a540c72e0 (ffff920a459526c8/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572161.104886:0:157593:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a51a38ab0 (ffff920a459526c8/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572161.104896:0:157593:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x203d:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788572161.104921:0:157593:0:(file.c:479:ll_file_release()) START file f73a.replay-single:[0x20001a9e3:0x203d:0x0](ffff920a50d2ba88), flags 100002 00000002:00100000:3.0:1788572161.104980:0:157593:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a46524a80 x1875444569452160/t330712481931(330712481931) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572163 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572161.104991:0:157593:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572161.104997:0:157593:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ea00 x1875444569466240/t0(0) o35->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572161.105025:0:157593:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:157593:1875444569466240:192.168.202.157@tcp:35:multiop.0 00000100:00100000:3.0:1788572161.105124:0:157593:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51a5e0c0 going to sleep for 11 seconds 00010000:00010000:3.0:1788572161.697653:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572163.746834:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572164.767419:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572166.647202:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572166.647246:0:33366:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584257408 00000100:00100000:3.0:1788572166.647256:0:33366:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584257408 00000100:00100000:3.0:1788572166.647316:0:33366:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9481 00000100:00100000:3.0:1788572166.647326:0:33366:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:177557:x1875444584257408:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572166.647357:0:33366:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff920a42f86800 lock: 0000000086415aae/0x8364c713e25bb1e5 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x19c2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x265cfe11109bcf18 expref: -99 pid: 157595 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572166.647446:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572166.647454:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:177557:x1875444584257408:12345-192.168.202.157@tcp:104: Request processed in 130us (263us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572166.647461:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9481 00000100:00100000:2.0:1788572167.645805:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569467264:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572167.645879:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569467392:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572167.645913:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569467648:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572167.645915:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569467520:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572167.645936:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569467776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572167.648937:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569467776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572167.649401:0:2399:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a42f86800: committing for last_committed 34359738469 gen 40 00000100:00100000:2.0:1788572167.649417:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569467648:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572167.652881:0:2399:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449090 gen 11 00000100:00100000:2.0:1788572167.652922:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569467392:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572167.653382:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569467520:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572167.653396:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569467264:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572167.838855:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572168.883200:0:159081:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 21:36:07 (1788572167) 00000080:00200000:2.0:1788572169.271406:0:159094:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572169.271441:0:159094:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572169.271463:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572169.271481:0:159094:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572169.271489:0:159094:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572169.271496:0:159094:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572169.271500:0:159094:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb21d 00000080:00010000:2.0:1788572169.271507:0:159094:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb21d from it ffffaf7300e2bbf8 00010000:00010000:2.0:1788572169.271511:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572169.271520:0:159094:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572169.271537:0:159094:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572169.271552:0:159094:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572169.271558:0:159094:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572169.271564:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572169.271578:0:159094:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:2.0:1788572169.271821:0:159094:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572169.271826:0:159094:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572169.271829:0:159094:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb21d 00000080:00010000:2.0:1788572169.271833:0:159094:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb21d from it ffffaf7300e2bb38 00010000:00010000:2.0:1788572169.271837:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572169.271844:0:159094:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:2.0:1788572169.271853:0:159094:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572169.271858:0:159094:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f73b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), file ffff920a43c1d680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572169.271866:0:159094:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f73b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), intent=open|creat 00000002:00010000:2.0:1788572169.271886:0:159094:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f73b.replay-single,[0x20001a9e3:0x203f:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572169.271898:0:159094:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572169.271957:0:159094:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572169.271965:0:159094:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572169.271968:0:159094:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a79000/0x8364c713e25bb21d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd228 expref: -99 pid: 158017 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:2.0:1788572169.272009:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.272016:0:159094:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572169.272028:0:159094:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572169.272052:0:159094:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891c00 x1875444569468032/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 664/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572169.272062:0:159094:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/0,1 mode: --/CW res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572169.272084:0:159094:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569468032:192.168.202.157@tcp:101:multiop.0 00000100:00100000:2.0:1788572169.272282:0:159094:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eeb0c0 going to sleep for 11 seconds 00010000:00010000:2.0:1788572169.290564:0:159094:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 4/0,1 mode: --/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290722:0:159094:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 4/0,1 mode: --/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290735:0:159094:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 4/0,1 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290749:0:159094:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 4/0,1 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290774:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 5/1,1 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290784:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 5/1,1 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290793:0:159094:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572169.290812:0:159094:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 664/664 e 0 to 0 dl 1788572185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572169.290827:0:159094:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 664/664 e 0 to 0 dl 1788572185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572169.290841:0:159094:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572169.290856:0:159094:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.290868:0:159094:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8255 (0 0) 00000080:00200000:2.0:1788572169.290876:0:159094:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572169.290889:0:159094:0:(namei.c:1628:ll_create_it()) VFS Op:name=f73b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), intent=open|creat 00000080:00200000:2.0:1788572169.290967:0:159094:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000080:00200000:2.0:1788572169.291070:0:159094:0:(file.c:6843:ll_layout_conf()) [0x20001a9e3:0x203f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572169.291076:0:159094:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000080:00010000:2.0:1788572169.291079:0:159094:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) for lock 0x8364c713e25bb239 00000080:00010000:2.0:1788572169.291093:0:159094:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000080:00200008:2.0:1788572169.291112:0:159094:0:(file.c:1090:ll_file_open()) START file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88), flags 100102 00000002:00100000:2.0:1788572169.291139:0:159094:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572185 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572169.291185:0:159094:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb239 from it ffff920a479e9a20 00010000:00010000:2.0:1788572169.291190:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572169.291200:0:159094:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572169.291220:0:159094:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88), inode mode 81a4 mask 40 00000080:00200000:3.0:1788572169.319637:0:159096:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572169.319666:0:159096:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572169.319730:0:159096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 159096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.319740:0:159096:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159096 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.319752:0:159096:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159096 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572169.319780:0:159096:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159096:1875444569468160:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572169.319865:0:159096:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572169.327242:0:159096:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159096:1875444569468160:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572169.327270:0:159096:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327284:0:159096:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327297:0:159096:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327309:0:159096:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327323:0:159096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327331:0:159096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327340:0:159096:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572169.327357:0:159096:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3c700 x1875444569468160/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572185 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572169.327372:0:159096:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327384:0:159096:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572169.327392:0:159096:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572169.327419:0:159096:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572169.327425:0:159096:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb240 00000080:00010000:3.0:1788572169.327430:0:159096:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572169.327434:0:159096:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb240 from it ffffaf7301d07b58 00010000:00010000:3.0:1788572169.327437:0:159096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327446:0:159096:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7a00/0x8364c713e25bb240 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd458 expref: -99 pid: 159096 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572169.327474:0:159096:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572169.327486:0:159096:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f73b.replay-single, flags=8 00000080:00200008:3.0:1788572169.327499:0:159096:0:(file.c:6296:ll_getattr_dentry()) START file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572169.327504:0:159096:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88),name=f73b.replay-single 00000002:00010000:3.0:1788572169.327524:0:159096:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x203f:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572169.327534:0:159096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 2/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572169.327546:0:159096:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 2/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572169.327552:0:159096:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572169.327557:0:159096:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000080:00010000:3.0:1788572169.327559:0:159096:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) for lock 0x8364c713e25bb239 00000080:00010000:3.0:1788572169.327563:0:159096:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb239 from it ffffaf7301d07cb8 00010000:00010000:3.0:1788572169.327565:0:159096:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572169.327572:0:159096:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816000/0x8364c713e25bb239 lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd443 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572169.327622:0:159096:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572169.327633:0:159096:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20001a9e3:0x203f:0x0] 00000008:00010000:3.0:1788572169.327656:0:159096:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572169.327659:0:159096:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff920a51a38230, osc lock ffff920a45802390, flags 0x400001002 00010000:00010000:3.0:1788572169.327708:0:159096:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.327718:0:159096:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572169.327760:0:159096:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:3.0:1788572169.327773:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572169.327790:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569468288:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572169.342369:0:2400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342403:0:2400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342418:0:2400:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342428:0:2400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:3.0:1788572169.342443:0:2400:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:3.0:1788572169.342455:0:2400:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:3.0:1788572169.342478:0:2400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342486:0:2400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342493:0:2400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572169.342502:0:2400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:3.0:1788572169.342530:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569468288:192.168.202.157@tcp:101:rm.0 00000080:00200000:2.0:1788572170.391471:0:159112:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572170.391504:0:159112:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572170.391525:0:159112:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572170.391542:0:159112:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572170.391551:0:159112:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572170.391558:0:159112:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572170.391562:0:159112:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb24e 00000080:00010000:2.0:1788572170.391569:0:159112:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb24e from it ffffaf730180bbd0 00010000:00010000:2.0:1788572170.391573:0:159112:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572170.391827:0:159112:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572170.391850:0:159112:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572170.391866:0:159112:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=1536 00000080:00200000:2.0:1788572170.391870:0:159112:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572170.391877:0:159112:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572170.391884:0:159112:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572170.391894:0:159112:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572170.391900:0:159112:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572170.391905:0:159112:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572170.391907:0:159112:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb24e 00000080:00010000:2.0:1788572170.391911:0:159112:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb24e from it ffffaf730180bcd0 00010000:00010000:2.0:1788572170.391914:0:159112:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572170.391922:0:159112:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572170.391930:0:159112:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572170.391935:0:159112:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572170.391940:0:159112:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572170.391945:0:159112:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572170.391958:0:159112:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572170.391965:0:159112:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572170.391968:0:159112:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572170.391970:0:159112:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb24e 00000080:00010000:2.0:1788572170.391973:0:159112:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb24e from it ffffaf730180bdb0 00010000:00010000:2.0:1788572170.391976:0:159112:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572170.391983:0:159112:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572170.391990:0:159112:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:2.0:1788572170.391995:0:159112:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=33188, excl=1 00000080:00200000:2.0:1788572170.391999:0:159112:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8) mode 100644 dev 0 00010000:00010000:2.0:1788572170.392033:0:159112:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572170.392083:0:159112:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572170.392090:0:159112:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572170.392093:0:159112:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70400/0x8364c713e25bb24e lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd482 expref: -99 pid: 159105 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000020:00100000:2.0:1788572170.392120:0:159112:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572170.392134:0:159112:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52892d80 x1875444569469440/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572170.392158:0:159112:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job mcreate:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159112:1875444569469440:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:2.0:1788572170.392377:0:159112:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eebb40 going to sleep for 11 seconds 00000080:00200000:3.0:1788572170.423886:0:159115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572170.423922:0:159115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572170.423999:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.424011:0:159115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.424033:0:159115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572170.424063:0:159115:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469568:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572170.424165:0:159115:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572170.427793:0:159115:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469568:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572170.427832:0:159115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427847:0:159115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427863:0:159115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427875:0:159115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427896:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427905:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427912:0:159115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572170.427925:0:159115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3c700 x1875444569469568/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572170.427940:0:159115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.427953:0:159115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572170.427959:0:159115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572170.427990:0:159115:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572170.427996:0:159115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb255 00000080:00010000:3.0:1788572170.428000:0:159115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572170.428004:0:159115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb255 from it ffffaf730180bb58 00010000:00010000:3.0:1788572170.428007:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.428015:0:159115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572170.428053:0:159115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572170.428075:0:159115:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=8 00000080:00200000:3.0:1788572170.428080:0:159115:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8), intent=getattr 00000002:00010000:3.0:1788572170.428104:0:159115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572170.428146:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.428153:0:159115:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.428163:0:159115:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 159115 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572170.428180:0:159115:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469696:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572170.429636:0:159115:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572170.439579:0:159115:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469696:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572170.439662:0:159115:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439686:0:159115:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439694:0:159115:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439707:0:159115:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439717:0:159115:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439730:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439738:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439745:0:159115:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572170.439757:0:159115:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3c700 x1875444569469696/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572170.439773:0:159115:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439784:0:159115:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8256 (0 0) 00000080:00200000:3.0:1788572170.439792:0:159115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572170.439819:0:159115:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x20001a9e3:0x2040:0x0], updating i_size 0 00000080:00200000:3.0:1788572170.439822:0:159115:0:(namei.c:147:ll_iget()) got inode: [0x20001a9e3:0x2040:0x0](ffff920a50d4aa08): rc = 0 00000080:00200000:3.0:1788572170.439844:0:159115:0:(file.c:6843:ll_layout_conf()) [0x20001a9e3:0x2040:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572170.439850:0:159115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x2040:0x0](ffff920a50d4aa08) for lock 0x8364c713e25bb25c 00000080:00010000:3.0:1788572170.439861:0:159115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x2040:0x0](ffff920a50d4aa08) 00000080:00010000:3.0:1788572170.439864:0:159115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb25c from it ffffaf730180bb20 00010000:00010000:3.0:1788572170.439868:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572170.439877:0:159115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572170.439922:0:159115:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x20001a9e3:0x2040:0x0](ffff920a50d4aa08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572170.439928:0:159115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x2040:0x0](ffff920a50d4aa08),name=fsa-oleg257-client.virtnet 00000002:00010000:3.0:1788572170.439948:0:159115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x2040:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572170.439957:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 2/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572170.439968:0:159115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 2/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572170.439974:0:159115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572170.439978:0:159115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x2040:0x0](ffff920a50d4aa08) 00000080:00010000:3.0:1788572170.439981:0:159115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x2040:0x0](ffff920a50d4aa08) for lock 0x8364c713e25bb25c 00000080:00010000:3.0:1788572170.439984:0:159115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb25c from it ffffaf730180bcb8 00010000:00010000:3.0:1788572170.439987:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572170.439994:0:159115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572170.440065:0:159115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572170.440073:0:159115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572170.440080:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572170.440089:0:159115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572170.440095:0:159115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572170.440099:0:159115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572170.440102:0:159115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb255 00000080:00010000:3.0:1788572170.440105:0:159115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb255 from it ffffaf730180bc10 00010000:00010000:3.0:1788572170.440108:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572170.440115:0:159115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572170.440123:0:159115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572170.440132:0:159115:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:3.0:1788572170.440137:0:159115:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572170.440143:0:159115:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572170.440149:0:159115:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572170.440158:0:159115:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572170.440164:0:159115:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572170.440168:0:159115:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572170.440171:0:159115:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb255 00000080:00010000:3.0:1788572170.440174:0:159115:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb255 from it ffffaf730180bd68 00010000:00010000:3.0:1788572170.440178:0:159115:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572170.440185:0:159115:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572170.440194:0:159115:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 3 00000080:00200000:3.0:1788572170.440199:0:159115:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a485780c8) 00010000:00010000:3.0:1788572170.440460:0:159115:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572170.440485:0:159115:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572170.440494:0:159115:0:(vvp_object.c:117:vvp_conf_set()) [0x20001a9e3:0x2040:0x0]: losing layout lock 00010000:00010000:3.0:1788572170.440528:0:159115:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 1/0,0 mode: --/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572170.440534:0:159115:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572170.440540:0:159115:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572170.440542:0:159115:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7c00/0x8364c713e25bb25c lrc: 0/0,0 mode: --/PR res: [0x20001a9e3:0x2040:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bd4ba expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572170.440560:0:159115:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7600/0x8364c713e25bb255 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bd49e expref: -99 pid: 159115 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572170.440577:0:159115:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572170.440618:0:159115:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3c700 x1875444569469952/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572170.440645:0:159115:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469952:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572170.440733:0:159115:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783fa80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572170.446047:0:159115:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159115:1875444569469952:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572170.446123:0:159115:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x2040:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572170.446136:0:159115:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x2040:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572170.446146:0:159115:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x2040:0x0](ffff920a50d4aa08) 00010000:00010000:3.0:1788572171.932656:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572176.027864:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572180.123924:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572183.013252:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569471488:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572183.013252:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569471360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572183.013336:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569471744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572183.013336:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569471616:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572183.013359:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569471872:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572183.016282:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569471744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572183.016346:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569471616:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572183.016358:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2400:1875444569471872:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572183.016507:0:2399:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff920a42f86800 of lustre-MDT0000_UUID@192.168.202.157@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788572183.016513:0:2399:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572183.016517:0:2399:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff920a42f86800: Connection to lustre-MDT0000 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1788572183.016523:0:2399:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572183.016528:0:2399:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8905 00000100:00080000:2.0:1788572183.016539:0:2399:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572183.016551:0:2399:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572183.016634:0:2399:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569472000/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572183.016667:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569471488:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572184.222307:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572188.123944:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572188.123953:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572188.123957:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572188.123967:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8941 00000100:00080000:3.0:1788572188.123977:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572188.123985:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572188.123998:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572188.124019:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3ed80 x1875444569472256/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572188.125428:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569472128:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.125502:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472512:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.125531:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572188.125537:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472384:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.125558:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472640:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.130538:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472384:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.130559:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472512:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572188.132885:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569472640:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572188.315808:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572192.415394:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572193.244506:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569472896:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572193.244538:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569473280:192.168.202.157@tcp:9:kworker.0 00000100:00080000:3.0:1788572193.273152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572193.273167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572193.273173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569472896:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572194.460527:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572196.526300:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572198.369042:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572198.369051:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572198.369055:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572198.369067:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8951 00000100:00080000:3.0:1788572198.369385:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572198.369395:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572198.369407:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572198.369430:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3ed80 x1875444569473536/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572198.369503:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569473408:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572198.369634:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569473792:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572198.369673:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572198.369679:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569473664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572198.373566:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569473664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572198.375631:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569473792:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572198.558075:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572199.387797:0:2400:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572183/real 1788572183] req@ffff920a51b3dc00 x1875444569471360/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572199 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572199.387835:0:2400:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a51b3dc00 x1875444569471360/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572199 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572199.387846:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788572199.387851:0:2400:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.157@tcp: Connection to MGS (at 192.168.202.157@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788572199.387857:0:2400:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.157@tcp for MGC192.168.202.157@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788572199.387859:0:2400:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788572199.387864:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a51b3dc00 x1875444569471360/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572199 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572199.387884:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a51b3d500 x1875444569472128/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572204 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572199.387899:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a528cd180 x1875444569472768/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572209 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572199.387924:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a51b3ce00 x1875444569473408/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572214 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788572199.387935:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572199.387938:0:2400:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572199.387939:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788572199.387942:0:2400:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:2.0:1788572199.387963:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569471360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572199.388373:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569473920:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572199.388421:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46525500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474048:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572199.388727:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569473408:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572199.388736:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569472128:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572199.402182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572199.402190:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572199.402194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474048:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572200.605354:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572202.651838:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572204.510094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474432:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572204.541718:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572204.541735:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572204.541739:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474432:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572204.699948:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572206.751627:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572208.795962:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572209.627792:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572199/real 1788572199] req@ffff920a46526d80 x1875444569473920/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572209.627835:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a46526d80 x1875444569473920/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572209 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572209.627860:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572209.627890:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572209.627895:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569473920:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572209.628664:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569474816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572209.631801:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474944:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572209.637957:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.157@tcp: connect to target with instance 0 10000000:01000000:3.0:1788572209.637977:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:3.0:1788572209.637985:0:2396:0:(import.c:1328:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.157@tcp) after server handle changed from 0x265cfe11109bd195 to 0x265cfe11109bd4f9 00000100:00080000:3.0:1788572209.637990:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:3.0:1788572209.637996:0:2396:0:(import.c:1718:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.157@tcp; invalidating 00000100:00100000:3.0:1788572209.640062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569474816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572209.643193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572209.643212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572209.643217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569474944:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572210.843952:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572211.634853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569475584:192.168.202.157@tcp:38:ll_cfg_requeue.0 00010000:00080000:3.0:1788572211.652298:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a528ce300 x1875444569475584/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572221 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572211.652315:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff920a42f86800: connect to target with instance 79 00000100:00080000:3.0:1788572211.652334:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:3.0:1788572211.652341:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572211.652344:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:3.0:1788572211.652350:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572211.652358:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 0 00000100:00080000:3.0:1788572211.652364:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572163 ref 1 fl New:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572211.652376:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569475584:192.168.202.157@tcp:38:ll_cfg_requeue.0 00000100:00100000:3.0:1788572211.652407:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444567597824:192.168.202.157@tcp:101:lfs.0 00000040:00080000:3.0:1788572211.698348:0:159431:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:3.0:1788572211.749982:0:15203:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:3.0:1788572211.750002:0:15203:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:3.0:1788572211.750012:0:15203:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572211.750025:0:15203:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572211.750034:0:15203:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:3.0:1788572211.750037:0:15203:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff920a51524d40 from 1 10000000:01000000:3.0:1788572211.750040:0:15203:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788572211.750078:0:15203:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572211.750087:0:15203:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572211.750098:0:15203:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572211.750125:0:15203:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:712ceec8-7c82-4602-bd77-3db1811a6b63:15203:1875444569476480:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:3.0:1788572211.750222:0:15203:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572211.755104:0:15203:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:712ceec8-7c82-4602-bd77-3db1811a6b63:15203:1875444569476480:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:3.0:1788572211.755138:0:15203:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572211.755156:0:15203:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572211.755244:0:15203:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:712ceec8-7c82-4602-bd77-3db1811a6b63:15203:1875444569476608:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:3.0:1788572211.755329:0:15203:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4783f840 going to sleep for 11 seconds 00010000:00010000:3.0:1788572212.892150:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572214.752555:0:15200:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572214.752572:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572214.752574:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572214.752575:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572215.964243:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572216.992018:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572218.012530:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572219.035853:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572220.059903:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572221.084787:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572222.114824:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572222.812254:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 0 seconds 00010000:00010000:3.0:1788572223.131984:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572223.837334:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 0 seconds 00010000:00010000:3.0:1788572224.157419:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572224.859843:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 0 seconds 00010000:00010000:3.0:1788572225.179829:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572225.885503:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 0 seconds 00010000:00010000:3.0:1788572226.204405:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572226.907711:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 0 seconds 00000100:00000400:2.0:1788572227.035806:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572211/real 1788572211] req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 1 dl 1788572227 ref 2 fl Rpc:XPQr/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572227.035844:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=REPLAY (now=REPLAY) req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 1 dl 1788572227 ref 2 fl Rpc:XPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00020000:2.0:1788572227.035863:0:2396:0:(client.c:3399:ptlrpc_replay_interpret()) @@@ request replay timed out req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 1 dl 1788572227 ref 2 fl Interpret:EXPQU/604/ffffffff rc -110/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572227.063736:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from REPLAY to CONNECTING 00000100:00080000:2.0:1788572227.063751:0:2396:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8970 00000100:00080000:2.0:1788572227.063766:0:2396:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a42f86800: import ffff920a5413a000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572227.063780:0:2396:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572227.063816:0:2396:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569478400/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572227.063836:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444567597824:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572227.063855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478400:192.168.202.157@tcp:38:ptlrpcd_rcv.0 00010000:00080000:2.0:1788572227.089036:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a52891500 x1875444569478400/t0(0) o38->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572237 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.089061:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff920a42f86800: connect to target with instance 79 00000100:00080000:2.0:1788572227.089082:0:2396:0:(import.c:1341:ptlrpc_connect_interpret()) reconnected to lustre-MDT0000_UUID@192.168.202.157@tcp after partition 00000100:00080000:2.0:1788572227.089085:0:2396:0:(import.c:1351:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff920a42f86800: reconnected to lustre-MDT0000_UUID during replay 00000100:00080000:2.0:1788572227.089088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788572227.089092:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572227.089095:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:2.0:1788572227.089102:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.089109:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 0 00000100:00080000:2.0:1788572227.089115:0:2396:0:(recover.c:118:ptlrpc_replay_next()) @@@ resend_replay=1, last_transno=0 req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 1 dl 1788572227 ref 1 fl Complete:EXPQU/606/ffffffff rc -110/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572227.089124:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 1 dl 1788572227 ref 1 fl New:EXPQU/606/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572227.089134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478400:192.168.202.157@tcp:38:ptlrpcd_rcv.0 00000100:00100000:2.0:1788572227.089171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444567597824:192.168.202.157@tcp:101:lfs.0 00000100:00080000:2.0:1788572227.134241:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/608 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788572227.134259:0:2396:0:(mdc_request.c:696:mdc_replay_open()) updating handle from 0x265cfe11109bd1db to 0x265cfe11109bd63b 00000100:00020000:2.0:1788572227.134264:0:2396:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/608 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572227.134273:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572227.134277:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:2.0:1788572227.134283:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.134292:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 326417519807 00000100:00080000:2.0:1788572227.134299:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572185 ref 1 fl New:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572227.134312:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444567597824:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572227.134334:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569468032:192.168.202.157@tcp:101:multiop.0 00000100:00080000:2.0:1788572227.150848:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788572227.150872:0:2396:0:(mdc_request.c:696:mdc_replay_open()) updating handle from 0x265cfe11109bd44a to 0x265cfe11109bd737 00000100:00020000:2.0:1788572227.150877:0:2396:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'multiop.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572227.150896:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572227.150899:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:2.0:1788572227.150905:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572186 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.150913:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 335007449091 00000100:00080000:2.0:1788572227.150919:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572186 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572227.150929:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569468032:192.168.202.157@tcp:101:multiop.0 00000100:00100000:2.0:1788572227.150955:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569469440:192.168.202.157@tcp:36:mcreate.0 00000100:00080000:2.0:1788572227.170032:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.170047:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572227.170049:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:2.0:1788572227.170054:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.170062:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 335007449093 00000100:00080000:2.0:1788572227.170069:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3c700 x1875444569469952/t335007449094(335007449094) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572186 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572227.170079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569469440:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:2.0:1788572227.170099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569469952:192.168.202.157@tcp:36:rm.0 00000100:00080000:2.0:1788572227.187863:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a51b3c700 x1875444569469952/t335007449094(335007449094) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572243 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.187889:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572227.187892:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449092 gen 11 00000100:00100000:2.0:1788572227.187899:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a52892d80 x1875444569469440/t335007449093(335007449093) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572243 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.187906:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a5413a000 from lustre-MDT0000_UUID committed 335007449092 last 335007449094 00000100:00080000:2.0:1788572227.187911:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788572227.188273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569469952:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572227.191189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478528:192.168.202.157@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788572227.191283:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46527100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478656:192.168.202.157@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788572227.198212:0:2396:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a60a78600/0x8364c713e25aed8a lrc: 2/0,0 mode: CR/CR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd761 expref: -99 pid: 153019 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:2.0:1788572227.198246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478528:192.168.202.157@tcp:101:ldlm_lock_repla.0 00010000:00010000:2.0:1788572227.198262:0:2396:0:(ldlm_request.c:2777:replay_lock_interpret()) ### replayed lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec6400/0x8364c713e25aecaa lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x1b8b:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd768 expref: -99 pid: 153007 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00080000:2.0:1788572227.198270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:2.0:1788572227.198301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478656:192.168.202.157@tcp:101:ldlm_lock_repla.0 00000100:00100000:2.0:1788572227.198331:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478784:192.168.202.157@tcp:400:ptlrpcd_rcv.0 00000100:00100000:2.0:1788572227.215565:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 335007449094 gen 11 00000100:00080000:2.0:1788572227.215632:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:2.0:1788572227.215646:0:2396:0:(client.c:3248:ptlrpc_resend_req()) @@@ going to resend req@ffff920a52891500 x1875444569478784/t0(0) o400->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572243 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.215657:0:2396:0:(client.c:3257:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff920a52891500 x1875444569478784/t0(0) o400->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572243 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ptlrpcd_rcv.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572227.215664:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from RECOVER to FULL 00000100:02000000:2.0:1788572227.217931:0:2396:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff920a42f86800: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00080000:2.0:1788572227.217946:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000005b19a6ef): req@ffff920a46527b80 x1875444569477504/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/1152 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572227.217980:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569478784:192.168.202.157@tcp:400:ptlrpcd_rcv.0 00000100:00100000:3.0:1788572227.218327:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d540 going to sleep for 11 seconds 00000100:00100000:2.0:1788572227.228243:0:159519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527b80 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569477504:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572227.228276:0:159519:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228291:0:159519:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228305:0:159519:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228318:0:159519:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228334:0:159519:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228345:0:159519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228353:0:159519:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572227.228367:0:159519:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a46527b80 x1875444569477504/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572243 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572227.228386:0:159519:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228402:0:159519:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572227.228408:0:159519:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572227.228437:0:159519:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1788572227.228443:0:159519:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:2.0:1788572227.228449:0:159519:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572227.228453:0:159519:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf73020ffcb8 00010000:00010000:2.0:1788572227.228456:0:159519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572227.228466:0:159519:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572227.232115:0:159519:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572227.232127:0:159519:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572227.232159:0:159519:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572227.232178:0:159519:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572227.232192:0:159519:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572227.232200:0:159519:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572227.232209:0:159519:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572227.232212:0:159519:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:2.0:1788572227.232219:0:159519:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf73020ffcb8 00010000:00010000:2.0:1788572227.232223:0:159519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572227.232233:0:159519:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572227.232399:0:159519:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572227.232407:0:159519:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572227.232415:0:159519:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572227.232424:0:159519:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572227.232430:0:159519:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572227.232434:0:159519:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572227.232436:0:159519:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:2.0:1788572227.232440:0:159519:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf73020ffbb0 00010000:00010000:2.0:1788572227.232443:0:159519:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572227.232451:0:159519:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572227.232459:0:159519:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 44 00000080:00200008:2.0:1788572227.232467:0:159519:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), flags 100000 00000080:00200080:2.0:1788572227.232576:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:2.0:1788572227.232616:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4c00 uarg=0000000000000000 00800000:00000004:2.0:1788572227.232623:0:159519:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: age 8928, max_age 8984 00000100:00100000:2.0:1788572227.232666:0:159519:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569478912:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572227.239929:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4805e180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572227.243570:0:159519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569478912:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572227.243629:0:159519:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: update ffff920a54025198 cache blocks 317241/353529 objects 1023637/1024000 00000080:00200080:3.0:1788572227.243718:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.243735:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00800000:00000004:3.0:1788572227.243743:0:159519:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a42f86800: age 8928, max_age 8984 00000100:00100000:3.0:1788572227.243779:0:159519:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569479040:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572227.243847:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48321780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572227.254900:0:159519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569479040:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572227.254939:0:159519:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a42f86800: update ffff920a540237d8 cache blocks 317340/353529 objects 1023725/1024000 00000080:00200080:3.0:1788572227.255033:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255057:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255067:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255072:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255077:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255081:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255085:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255090:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255094:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255098:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255102:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255106:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255111:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255115:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255119:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255123:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255127:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255131:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255136:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255141:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255161:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255166:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255171:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255175:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255179:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255183:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255187:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255191:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255195:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255198:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255202:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255206:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255210:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255214:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255218:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255221:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255225:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255229:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255233:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255236:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255240:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255244:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255248:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255251:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255256:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255260:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255264:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255268:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255272:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255275:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255279:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255283:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255286:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255290:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255294:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255297:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255301:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255304:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255308:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255311:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255315:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255318:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255322:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255325:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255329:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255345:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255350:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255354:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255358:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255361:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255366:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255370:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255374:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255377:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255381:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255384:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255388:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255392:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255395:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255399:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255403:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255406:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255410:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255414:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255418:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255421:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255425:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255428:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255433:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255436:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255439:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255443:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255446:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255450:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255454:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255458:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255461:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255465:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255469:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255472:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255476:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255479:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255483:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255487:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255490:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.255494:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.255697:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257494:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257813:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257820:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257824:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257828:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257831:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257834:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257837:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257840:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257843:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257847:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257850:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257854:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257859:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257863:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257866:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257870:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257874:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00800000:00000080:3.0:1788572227.257877:0:159519:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.257881:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.257889:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00020000:00000004:3.0:1788572227.257896:0:159519:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a42f86800: age 8928, max_age 8984 00000100:00100000:3.0:1788572227.257942:0:159519:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569479168:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572227.258035:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48321780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572227.272652:0:159519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569479168:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572227.272686:0:159519:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a42f86800: update ffff920a515237d8 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:3.0:1788572227.272770:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.272791:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00020000:00000004:3.0:1788572227.272800:0:159519:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a42f86800: age 8928, max_age 8984 00000100:00080000:3.0:1788572227.272814:0:159519:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a42f86800: connect to NID 192.168.202.157@tcp last attempt 8905 00000100:00080000:3.0:1788572227.272826:0:159519:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a42f86800: import ffff920a45969800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572227.272836:0:159519:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:3.0:1788572227.272861:0:159519:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3e680 x1875444569479296/t0(0) o8->lustre-OST0001-osc-ffff920a42f86800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572227.272904:0:159519:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a51b3ce00 x1875444569479424/t0(0) o13->lustre-OST0001-osc-ffff920a42f86800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572227.272913:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48321780 going to sleep for 0 seconds 00000100:00100000:2.0:1788572227.273218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569479296:192.168.202.157@tcp:8:lfs.0 00000100:00100000:3.0:1788572227.279936:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff920a42f86800: committing for last_committed 4294975670 gen 36 00000100:00080000:3.0:1788572227.279957:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff920a42f86800: connect to target with instance 1 00000100:00080000:3.0:1788572227.279967:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff920a42f86800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:3.0:1788572227.279980:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:3.0:1788572227.279984:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a45969800 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:3.0:1788572227.279994:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:3.0:1788572227.280001:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:3.0:1788572227.280015:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000641acbf7): req@ffff920a51b3ce00 x1875444569479424/t0(0) o13->lustre-OST0001-osc-ffff920a42f86800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572227.280041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569479296:192.168.202.157@tcp:8:lfs.0 00000100:00100000:2.0:1788572227.280195:0:159519:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48321780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572227.306297:0:159519:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159519:1875444569479424:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572227.306328:0:159519:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a42f86800: update ffff920a51521e18 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:3.0:1788572227.306414:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306433:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306439:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306442:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306446:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306449:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306452:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306454:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306457:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306460:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306463:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306465:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306468:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306472:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306475:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306478:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306481:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306484:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306487:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306490:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306492:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306495:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306498:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306501:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306503:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306506:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306509:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306512:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306514:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306517:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306520:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306522:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306525:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306527:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306530:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306533:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306536:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306538:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306542:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306544:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306548:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306551:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306553:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306556:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306559:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306561:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306564:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306566:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306569:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306572:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306575:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306577:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306580:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306583:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306627:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306631:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306634:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306636:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306639:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306642:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306645:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306647:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306650:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306654:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306657:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306660:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306662:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306665:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306668:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306670:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306673:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306676:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306679:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306681:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306684:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306686:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306689:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306699:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306703:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306706:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306709:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306711:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306714:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306716:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306719:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306722:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306725:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306728:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306731:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306734:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306738:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306740:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306743:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306746:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306749:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306751:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306754:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306757:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306760:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306762:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306765:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306767:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306770:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306773:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306776:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306778:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306781:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306783:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306786:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306789:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306792:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306794:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306797:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306800:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306803:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306805:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306808:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306811:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306814:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306817:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306820:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306823:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306827:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306830:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200080:3.0:1788572227.306833:0:159519:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffd0ef831d0 00020000:00000080:3.0:1788572227.306835:0:159519:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529bc800 uarg=0000000000000000 00000080:00200008:3.0:1788572227.306849:0:159519:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), flags 100000 00010000:00010000:2.0:1788572228.255217:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572229.275871:0:33367:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572230.305497:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572232.354165:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572234.396778:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572236.444807:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572237.275857:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480320:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.275945:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480576:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.275977:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572237.275983:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569480192:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.276015:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480448:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.276038:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480704:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.282645:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480704:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572237.282666:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480576:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572237.287553:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:712ceec8-7c82-4602-bd77-3db1811a6b63:2399:1875444569480192:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572237.287579:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480448:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572237.287655:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2399:1875444569480320:192.168.202.157@tcp:400:kworker.0 00000001:02000400:2.0:1788572237.507119:0:159951:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:3.0:1788572238.491992:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572239.515918:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572239.826857:0:159989:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524380 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569480832:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572239.826891:0:159989:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a42f86800: update ffff920a54025198 cache blocks 317241/353529 objects 1023638/1024000 00000080:00200080:3.0:1788572239.826950:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.826964:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00800000:00000004:3.0:1788572239.826971:0:159989:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a42f86800: age 8985, max_age 8997 00000100:00100000:3.0:1788572239.827004:0:159989:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569480960:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572239.827067:0:159989:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba99c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.828806:0:159989:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569480960:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572239.828832:0:159989:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a42f86800: update ffff920a540237d8 cache blocks 317340/353529 objects 1023725/1024000 00000080:00200080:3.0:1788572239.828887:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828902:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828910:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828914:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828918:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828922:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828926:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828930:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828933:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828937:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828940:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828944:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828948:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828952:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828956:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828960:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828964:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828967:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828972:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828977:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828981:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828985:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828989:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.828992:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.828996:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829000:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829004:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829007:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829012:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829015:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829029:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829033:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829038:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829041:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829045:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829048:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829052:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829056:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829060:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829063:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829067:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829071:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829075:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829078:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829082:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829086:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829090:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829093:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829097:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829100:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829105:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829108:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829112:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829114:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829118:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829121:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829125:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829128:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829131:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829134:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829138:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829141:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829146:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829149:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829154:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829157:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829162:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829165:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829168:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829171:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829175:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829178:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829181:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829184:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829188:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829191:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829195:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829199:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829203:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829221:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829226:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829230:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829234:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829237:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829241:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829244:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829247:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829250:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829253:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829256:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829259:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829263:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829266:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829269:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829273:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829276:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829279:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829282:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829285:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829288:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829291:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829294:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829297:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829300:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829304:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829306:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829310:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829313:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829316:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829319:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829322:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829325:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829328:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829331:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829334:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829337:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829341:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829344:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829347:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829350:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829353:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829355:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829359:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829362:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829365:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00800000:00000080:3.0:1788572239.829367:0:159989:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00000080:00200080:3.0:1788572239.829371:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:3.0:1788572239.829377:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00020000:00000004:3.0:1788572239.829385:0:159989:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a42f86800: age 8985, max_age 8997 00000100:00100000:3.0:1788572239.829418:0:159989:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569481088:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572239.829476:0:159989:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba99c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.837347:0:159989:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569481088:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572239.837376:0:159989:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a42f86800: update ffff920a515237d8 cache blocks 901356/958279 objects 261710/262144 00000080:00200080:3.0:1788572239.837450:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:3.0:1788572239.837466:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a529be000 uarg=0000000000000000 00020000:00000004:3.0:1788572239.837473:0:159989:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a42f86800: age 8985, max_age 8997 00000100:00100000:3.0:1788572239.837503:0:159989:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569481216:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572239.837564:0:159989:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba99c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.847683:0:159989:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job lfs:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159989:1875444569481216:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572239.847727:0:159989:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a42f86800: update ffff920a51521e18 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:2.0:1788572239.847897:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847923:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847932:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847936:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847941:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847944:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847948:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847954:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847959:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847963:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847968:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847973:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847977:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847981:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847987:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847990:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.847995:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.847998:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848003:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848006:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848011:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848014:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848030:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848035:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848039:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848044:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848049:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848054:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848059:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848064:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848068:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848074:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848078:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848082:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848086:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848089:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848093:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848097:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848101:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848106:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848110:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848113:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848117:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848122:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848126:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848131:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848136:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848141:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848145:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848148:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848152:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848156:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848160:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848165:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848170:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848174:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848178:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848183:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848188:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848192:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848198:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848203:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848450:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848459:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848464:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848468:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848473:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848478:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848483:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848488:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848494:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848498:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848503:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848506:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848510:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848514:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848519:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848522:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848526:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848530:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848535:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848539:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848543:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848547:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848552:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848556:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848560:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848563:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848567:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848570:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848575:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848578:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848582:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848630:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848636:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848639:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848644:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848648:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848652:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848656:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848659:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848663:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848667:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848670:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848674:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848678:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848682:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848685:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848689:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848692:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848696:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848699:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848703:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848707:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848711:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848714:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848718:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848721:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848725:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848728:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848732:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848736:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848740:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848744:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200080:2.0:1788572239.848748:0:159989:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) cmd=c00866a4 arg=7ffdaa468070 00020000:00000080:2.0:1788572239.848751:0:159989:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a42f86800: cmd=c00866a4 len=592 karg=ffff920a46ef4000 uarg=0000000000000000 00000080:00200008:2.0:1788572239.848769:0:159989:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), flags 100000 00000080:00200008:3.0:1788572239.877487:0:159094:0:(llite_lib.c:2427:ll_setattr_raw()) START file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) current size 0, valid attrs 41, mode 8000, uid 1, gid 0, new size 18446744071817554661, atime 1788572239.000000000, mtime 1788572239.000000000, ctime 1788572239.000000000 00000020:00100000:3.0:1788572239.877622:0:159094:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572239.877638:0:159094:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ce00 x1875444569481472/t0(0) o36->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 488/512 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572239.877670:0:159094:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569481472:192.168.202.157@tcp:36:multiop.0 00000100:00100000:3.0:1788572239.877788:0:159094:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba99c0 going to sleep for 11 seconds 00000080:00200000:3.0:1788572239.915154:0:159094:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 0/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788572239.915174:0:159094:0:(llite_lib.c:2686:ll_setattr_raw()) COMPLETED file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) current size 0, valid attrs 10000041, mode 8000, uid 1, gid 0, new size 18446744071817554661, atime 1788572239.000000000, mtime 1788572239.000000000, ctime 1788572239.000000000, rc 0 00000080:00200008:3.0:1788572239.915205:0:159094:0:(file.c:6296:ll_getattr_dentry()) START file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572239.915213:0:159094:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88),name=f73b.replay-single 00000002:00010000:3.0:1788572239.915233:0:159094:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x20001a9e3:0x203f:0x0]) in obj [0x20001a9e3:0x203f:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572239.915248:0:159094:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 30 res 8590043619/8255 (0 0) 00010000:00010000:3.0:1788572239.915299:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.915311:0:159094:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.915321:0:159094:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: --/CR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 159094 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572239.915345:0:159094:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569481856:192.168.202.157@tcp:101:multiop.0 00000100:00100000:3.0:1788572239.915415:0:159094:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba9780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.925479:0:159094:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569481856:192.168.202.157@tcp:101:multiop.0 00010000:00010000:3.0:1788572239.925509:0:159094:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 4/1,0 mode: --/CR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925523:0:159094:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 4/1,0 mode: --/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925537:0:159094:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925549:0:159094:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925563:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925571:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 5/2,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925579:0:159094:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 4/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572239.925607:0:159094:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3e680 x1875444569481856/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 576/640 e 0 to 0 dl 1788572255 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'multiop.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572239.925619:0:159094:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572239.925633:0:159094:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925648:0:159094:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a45851000 type 13 mode 4 res 8590043619/8255 (0 0) 00000080:00200000:3.0:1788572239.925654:0:159094:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x20001a9e3:0x203f:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572239.925706:0:159094:0:(file.c:6843:ll_layout_conf()) [0x20001a9e3:0x203f:0x0]: layout version change: 0 -> 0 00000080:00010000:3.0:1788572239.925711:0:159094:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) for lock 0x8364c713e25bb28d 00000080:00010000:3.0:1788572239.925717:0:159094:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000080:00010000:3.0:1788572239.925720:0:159094:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb28d from it ffffaf7300e2bcd8 00010000:00010000:3.0:1788572239.925723:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 3/1,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.925731:0:159094:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a46ec7400/0x8364c713e25bb28d lrc: 2/0,0 mode: PR/PR res: [0x20001a9e3:0x203f:0x0].0x0 bits 0x1a/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bd952 expref: -99 pid: 159094 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572239.925768:0:159094:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:3.0:1788572239.925778:0:159094:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x20001a9e3:0x203f:0x0] 00000008:00010000:3.0:1788572239.925797:0:159094:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:3.0:1788572239.925801:0:159094:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff920a51a38d30, osc lock ffff920a458031d0, flags 0x400001002 00010000:00010000:3.0:1788572239.925817:0:159094:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572239.925832:0:159094:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 2/1,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000020:00200000:3.0:1788572239.925844:0:159094:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:3.0:1788572239.925848:0:159094:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572239.925861:0:159094:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000020:00200000:3.0:1788572239.925872:0:159094:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:3.0:1788572239.925882:0:159094:0:(file.c:1775:ll_merge_attr_nolock()) [0x20001a9e3:0x203f:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788572239.925887:0:159094:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff920a465284a0 (ffff920a529b4368/0) at cl_lock_release():245 00000020:00010000:3.0:1788572239.925890:0:159094:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a465284a0 (ffff920a529b4368/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572239.925893:0:159094:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a51a38d30 (ffff920a529b4d88/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572239.925896:0:159094:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a465284a0 (ffff920a529b4368/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572239.925899:0:159094:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a51a38d30 (ffff920a529b4368/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572239.925908:0:159094:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200008:3.0:1788572239.925931:0:159094:0:(file.c:479:ll_file_release()) START file f73b.replay-single:[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88), flags 100002 00000002:00100000:3.0:1788572239.925989:0:159094:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572243 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572239.926000:0:159094:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572239.926005:0:159094:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3e680 x1875444569482112/t0(0) o35->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572239.926054:0:159094:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569482112:192.168.202.157@tcp:35:multiop.0 00000100:00100000:3.0:1788572239.926134:0:159094:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba9780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.931722:0:159094:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job multiop:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:159094:1875444569482112:192.168.202.157@tcp:35:multiop.0 00000002:00100000:3.0:1788572239.931766:0:159094:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a52891c00 x1875444569468032/t335007449091(335007449091) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/608 e 0 to 0 dl 1788572243 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'multiop.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572239.931816:0:159094:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572239.931831:0:159094:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572239.931860:0:159094:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x203f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572239.931870:0:159094:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x203f:0x0](ffff920a50d2ba88) 00000008:00010000:3.0:1788572239.931905:0:159094:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 0 0 0 by oinfo size 0 blocks 0 [cma]time 1788572239 0 0 ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572239.933550:0:158949:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572239.933577:0:158949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572239.933618:0:158949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572239.933634:0:158949:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788572239.933643:0:158949:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572239.933651:0:158949:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572239.933655:0:158949:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:2.0:1788572239.933663:0:158949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf7301c37b58 00010000:00010000:2.0:1788572239.933667:0:158949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572239.933676:0:158949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572239.933691:0:158949:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572239.933706:0:158949:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f73b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), flags=9 00000080:00200000:2.0:1788572239.933710:0:158949:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f73b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a485780c8), intent=getattr 00000002:00010000:2.0:1788572239.933725:0:158949:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f73b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:2.0:1788572239.933768:0:158949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572239.933776:0:158949:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572239.933784:0:158949:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572239.933810:0:158949:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job bash:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:158949:1875444569482240:192.168.202.157@tcp:101:bash.0 00000100:00100000:2.0:1788572239.933891:0:158949:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a479ed300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572239.938922:0:158949:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job bash:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:158949:1875444569482240:192.168.202.157@tcp:101:bash.0 00010000:00010000:3.0:1788572239.938948:0:158949:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.938976:0:158949:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.938983:0:158949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.938992:0:158949:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939001:0:158949:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939007:0:158949:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572239.939011:0:158949:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939028:0:158949:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939036:0:158949:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939045:0:158949:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572239.939052:0:158949:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572239.939059:0:158949:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a47816400/0x8364c713e25bb294 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 158949 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572239.939077:0:158949:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a52892680 x1875444569482240/t0(0) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572255 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'bash.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572239.939091:0:158949:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572239.939120:0:158949:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572239.939133:0:158949:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00010000:3.0:1788572239.939144:0:158949:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf7301c37930 00010000:00010000:3.0:1788572239.939148:0:158949:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572239.939155:0:158949:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572240.541028:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572242.589309:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572243.611828:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572244.638087:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572245.503396:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572245.503447:0:33366:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584298368 00000100:00100000:3.0:1788572245.503464:0:33366:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584298368 00000100:00100000:3.0:1788572245.503515:0:33366:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9482 00000100:00100000:3.0:1788572245.503525:0:33366:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8431:x1875444584298368:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572245.503547:0:33366:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff920a42f86800 lock: 000000006dee0445/0x8364c713e25bb247 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x19e2:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x265cfe11109bd45f expref: -99 pid: 159096 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572245.523202:0:33366:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572245.523229:0:33366:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_002:LOV_OSC_UUID+4:8431:x1875444584298368:12345-192.168.202.157@tcp:104: Request processed in 19695us (19836us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572245.523244:0:33366:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9482 00010000:00010000:3.0:1788572245.662008:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572246.695538:0:33368:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572247.688557:0:160579:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 21:37:26 (1788572246) 00010000:00010000:2.0:1788572247.709845:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572248.107027:0:160597:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572248.107041:0:160597:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:2.0:1788572248.107067:0:160597:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572248.107087:0:160597:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572248.107102:0:160597:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572248.107110:0:160597:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572248.107116:0:160597:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:2.0:1788572248.107120:0:160597:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:2.0:1788572248.107127:0:160597:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf7300da3cb8 00010000:00010000:2.0:1788572248.107131:0:160597:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:2.0:1788572248.107214:0:160597:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:2.0:1788572249.756540:0:15200:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572250.073534:0:160601:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572250.073547:0:160601:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572250.073568:0:160601:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572250.073760:0:160601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572250.073778:0:160601:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572250.073788:0:160601:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572250.073796:0:160601:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572250.073800:0:160601:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:3.0:1788572250.073807:0:160601:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf7301d6fcb8 00010000:00010000:3.0:1788572250.073812:0:160601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572250.073821:0:160601:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200008:3.0:1788572250.074047:0:160601:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a485780c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572250.074052:0:160601:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8),name=/ 00000002:00010000:3.0:1788572250.074062:0:160601:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572250.074081:0:160601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572250.074091:0:160601:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572250.074099:0:160601:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572250.074104:0:160601:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00010000:3.0:1788572250.074107:0:160601:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a485780c8) for lock 0x8364c713e25bb27f 00000080:00010000:3.0:1788572250.074111:0:160601:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb27f from it ffffaf7301d6fcb8 00010000:00010000:3.0:1788572250.074114:0:160601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572250.074238:0:160601:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a42f86800 lock: ffff920a51a70600/0x8364c713e25bb27f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bd776 expref: -99 pid: 159519 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572250.098557:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a485780c8) 00000080:00200000:3.0:1788572250.098636:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.098649:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.098690:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.098695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x9:0x0](ffff920a48524b08) 00000080:00200000:3.0:1788572250.098748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0xb:0x0](ffff920a4857e3c8) 00000080:00200000:3.0:1788572250.098757:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200003ab0:0x1:0x0](ffff920a4857f448) 00000080:00200000:3.0:1788572250.098766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000402:0x1:0x0](ffff920a4857b248) 00000080:00200000:3.0:1788572250.098782:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.098786:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.098790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0xc:0x0](ffff920a48573248) 00000080:00200000:3.0:1788572250.098805:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.098810:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.098818:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.098821:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x15:0x0](ffff920a485c1148) 00000080:00200000:3.0:1788572250.098845:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.098850:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.098857:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.098861:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x16:0x0](ffff920a48521148) 00000080:00200000:3.0:1788572250.098887:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.098891:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.098897:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.098901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x17:0x0](ffff920a4857d348) 00000080:00200000:3.0:1788572250.098920:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.098925:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103324:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x18:0x0](ffff920a4857ec08) 00000080:00200000:3.0:1788572250.103449:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103459:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103473:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x19:0x0](ffff920a485700c8) 00000080:00200000:3.0:1788572250.103511:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103516:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103524:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103527:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x1a:0x0](ffff920a4857db88) 00000080:00200000:3.0:1788572250.103551:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103556:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103563:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x1b:0x0](ffff920a4857c2c8) 00000080:00200000:3.0:1788572250.103614:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103620:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103627:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103632:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x1c:0x0](ffff920a48578908) 00000080:00200000:3.0:1788572250.103656:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103661:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103668:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103673:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x1d:0x0](ffff920a485c3a88) 00000080:00200000:3.0:1788572250.103696:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103702:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103708:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103713:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x1e:0x0](ffff920a485763c8) 00000080:00200000:3.0:1788572250.103735:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x107:0x0](ffff920a48657448) 00000080:00200000:3.0:1788572250.103749:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x109:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.103755:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x109:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103758:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x109:0x0](ffff920a48653248) 00000080:00200000:3.0:1788572250.103771:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x115:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103776:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103783:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x115:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x115:0x0](ffff920a485c63c8) 00000080:00200000:3.0:1788572250.103810:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x118:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103815:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103822:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x118:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103826:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x118:0x0](ffff920a48652a08) 00000080:00200000:3.0:1788572250.103847:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x122:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103852:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103858:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x122:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x122:0x0](ffff920a486521c8) 00000080:00200000:3.0:1788572250.103907:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x123:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103913:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103921:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x123:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103925:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x123:0x0](ffff920a4865b248) 00000080:00200000:3.0:1788572250.103962:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200001b71:0x12e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.103968:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.103975:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200001b71:0x12e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.103980:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200001b71:0x12e:0x0](ffff920a486542c8) 00000080:00200000:3.0:1788572250.104003:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe01:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104008:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104015:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe01:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104020:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe01:0x1:0x0](ffff920a4865e3c8) 00000080:00200000:3.0:1788572250.104043:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe02:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104048:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104055:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe02:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104059:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe02:0x3:0x0](ffff920a4865cb08) 00000080:00200000:3.0:1788572250.104105:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe02:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104111:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104363:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe02:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104368:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe02:0x4:0x0](ffff920a485c00c8) 00000080:00200000:3.0:1788572250.104465:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe02:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104472:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104480:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe02:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe02:0x9:0x0](ffff920a48653a88) 00000080:00200000:3.0:1788572250.104509:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe02:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104514:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104521:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe02:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104524:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe02:0xa:0x0](ffff920a48659148) 00000080:00200000:3.0:1788572250.104548:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20000fe02:0x21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104553:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104560:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20000fe02:0x21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20000fe02:0x21:0x0](ffff920a4865ec08) 00000080:00200000:3.0:1788572250.104643:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x200017331:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104649:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104657:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x200017331:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104662:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200017331:0x1:0x0](ffff920a485c3248) 00000080:00200000:3.0:1788572250.104688:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200017b01:0x1:0x0](ffff920a486563c8) 00000080:00200000:3.0:1788572250.104702:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x643:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.104708:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.104716:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x643:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104719:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x643:0x0](ffff920a47e83a88) 00000080:00200000:3.0:1788572250.104743:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x968:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104748:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x968:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x968:0x0](ffff920a487880c8) 00000080:00200000:3.0:1788572250.104767:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x969:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104771:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x969:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104774:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x969:0x0](ffff920a48633a88) 00000080:00200000:3.0:1788572250.104787:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104791:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104795:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96a:0x0](ffff920a4878e3c8) 00000080:00200000:3.0:1788572250.104807:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104811:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96b:0x0](ffff920a4878cb08) 00000080:00200000:3.0:1788572250.104827:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104831:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104834:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96c:0x0](ffff920a48781988) 00000080:00200000:3.0:1788572250.104847:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104850:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96d:0x0](ffff920a487800c8) 00000080:00200000:3.0:1788572250.104865:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104869:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96e:0x0](ffff920a48789988) 00000080:00200000:3.0:1788572250.104885:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x96f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104888:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x96f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x96f:0x0](ffff920a4878a1c8) 00000080:00200000:3.0:1788572250.104904:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x970:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104908:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x970:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x970:0x0](ffff920a48789148) 00000080:00200000:3.0:1788572250.104925:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x971:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104940:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x971:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104944:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x971:0x0](ffff920a4878ba88) 00000080:00200000:3.0:1788572250.104958:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x972:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104962:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x972:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x972:0x0](ffff920a48790908) 00000080:00200000:3.0:1788572250.104979:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x973:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.104983:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x973:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.104986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x973:0x0](ffff920a48788908) 00000080:00200000:3.0:1788572250.104999:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x974:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105002:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x974:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x974:0x0](ffff920a48787448) 00000080:00200000:3.0:1788572250.105015:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x975:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105019:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x975:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105023:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x975:0x0](ffff920a48783248) 00000080:00200000:3.0:1788572250.105034:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x976:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105037:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x976:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105041:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x976:0x0](ffff920a4878c2c8) 00000080:00200000:3.0:1788572250.105053:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x977:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105057:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x977:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105183:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x977:0x0](ffff920a4878f448) 00000080:00200000:3.0:1788572250.105204:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x978:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105208:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x978:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105212:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x978:0x0](ffff920a48795b88) 00000080:00200000:3.0:1788572250.105225:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x979:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105229:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x979:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105233:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x979:0x0](ffff920a48631148) 00000080:00200000:3.0:1788572250.105246:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105250:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105253:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97a:0x0](ffff920a48635b88) 00000080:00200000:3.0:1788572250.105266:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105270:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97b:0x0](ffff920a48636c08) 00000080:00200000:3.0:1788572250.105285:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105289:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105293:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97c:0x0](ffff920a48785348) 00000080:00200000:3.0:1788572250.105306:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105310:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97d:0x0](ffff920a487842c8) 00000080:00200000:3.0:1788572250.105327:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105331:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97e:0x0](ffff920a48780908) 00000080:00200000:3.0:1788572250.105346:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x97f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105350:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x97f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105354:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x97f:0x0](ffff920a487821c8) 00000080:00200000:3.0:1788572250.105368:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x980:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105372:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x980:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105376:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x980:0x0](ffff920a48781148) 00000080:00200000:3.0:1788572250.105513:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x981:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105518:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x981:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105522:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x981:0x0](ffff920a48783a88) 00000080:00200000:3.0:1788572250.105533:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x982:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105537:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x982:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x982:0x0](ffff920a4878d348) 00000080:00200000:3.0:1788572250.105555:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x983:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105558:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x983:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x983:0x0](ffff920a4878ec08) 00000080:00200000:3.0:1788572250.105574:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x984:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105577:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x984:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x984:0x0](ffff920a4878db88) 00000080:00200000:3.0:1788572250.105701:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x985:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105706:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x985:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105710:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x985:0x0](ffff920a4878aa08) 00000080:00200000:3.0:1788572250.105735:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x986:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105739:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x986:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x986:0x0](ffff920a4878b248) 00000080:00200000:3.0:1788572250.105757:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x987:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105761:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x987:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105765:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x987:0x0](ffff920a47ebdb88) 00000080:00200000:3.0:1788572250.105778:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x988:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105782:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x988:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x988:0x0](ffff920a47ebaa08) 00000080:00200000:3.0:1788572250.105798:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x989:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105801:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x989:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x989:0x0](ffff920a47ebc2c8) 00000080:00200000:3.0:1788572250.105818:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105822:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98a:0x0](ffff920a48784b08) 00000080:00200000:3.0:1788572250.105837:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105841:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105845:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98b:0x0](ffff920a487863c8) 00000080:00200000:3.0:1788572250.105857:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105861:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105865:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98c:0x0](ffff920a48786c08) 00000080:00200000:3.0:1788572250.105876:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105879:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105883:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98d:0x0](ffff920a48785b88) 00000080:00200000:3.0:1788572250.105896:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105900:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105904:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98e:0x0](ffff920a47eb8908) 00000080:00200000:3.0:1788572250.105917:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x98f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105921:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x98f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105925:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x98f:0x0](ffff920a47ebba88) 00000080:00200000:3.0:1788572250.105952:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x990:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105956:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x990:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105959:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x990:0x0](ffff920a47eb9148) 00000080:00200000:3.0:1788572250.105973:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x991:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105976:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x991:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105979:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x991:0x0](ffff920a48782a08) 00000080:00200000:3.0:1788572250.105991:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x992:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.105995:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x992:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.105999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x992:0x0](ffff920a47e93a88) 00000080:00200000:3.0:1788572250.106011:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x993:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106015:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x993:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x993:0x0](ffff920a47eba1c8) 00000080:00200000:3.0:1788572250.106031:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x994:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106036:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x994:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x994:0x0](ffff920a47eb9988) 00000080:00200000:3.0:1788572250.106053:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x995:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106057:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x995:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x995:0x0](ffff920a47e91148) 00000080:00200000:3.0:1788572250.106483:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x996:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106489:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x996:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x996:0x0](ffff920a47e921c8) 00000080:00200000:3.0:1788572250.106509:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x997:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106513:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x997:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x997:0x0](ffff920a47e91988) 00000080:00200000:3.0:1788572250.106529:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x998:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106534:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x998:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106537:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x998:0x0](ffff920a47ebcb08) 00000080:00200000:3.0:1788572250.106551:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x999:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106556:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x999:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x999:0x0](ffff920a47ebe3c8) 00000080:00200000:3.0:1788572250.106573:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106577:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106580:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99a:0x0](ffff920a47eb80c8) 00000080:00200000:3.0:1788572250.106720:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106726:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106730:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99b:0x0](ffff920a47ebf448) 00000080:00200000:3.0:1788572250.106745:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106749:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106753:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99c:0x0](ffff920a47ebb248) 00000080:00200000:3.0:1788572250.106767:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106771:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99d:0x0](ffff920a47ebd348) 00000080:00200000:3.0:1788572250.106789:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106793:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106797:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99e:0x0](ffff920a47ebec08) 00000080:00200000:3.0:1788572250.106811:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x99f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106815:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x99f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x99f:0x0](ffff920a47eb6c08) 00000080:00200000:3.0:1788572250.106832:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106837:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a0:0x0](ffff920a48635348) 00000080:00200000:3.0:1788572250.106856:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106859:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106863:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a1:0x0](ffff920a47eb5b88) 00000080:00200000:3.0:1788572250.106876:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106880:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a2:0x0](ffff920a47eb2a08) 00000080:00200000:3.0:1788572250.106898:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106902:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a3:0x0](ffff920a48796c08) 00000080:00200000:3.0:1788572250.106920:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106923:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106927:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a4:0x0](ffff920a47eb42c8) 00000080:00200000:3.0:1788572250.106953:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106957:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106962:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a5:0x0](ffff920a47eb0908) 00000080:00200000:3.0:1788572250.106976:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.106980:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.106984:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a6:0x0](ffff920a47eb3a88) 00000080:00200000:3.0:1788572250.106997:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.107001:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.107004:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a7:0x0](ffff920a47e94b08) 00000080:00200000:3.0:1788572250.107019:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.107023:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.107027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a8:0x0](ffff920a47e963c8) 00000080:00200000:3.0:1788572250.107040:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9a9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.107044:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9a9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.107047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9a9:0x0](ffff920a47e900c8) 00000080:00200000:3.0:1788572250.107060:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9aa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.107064:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9aa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108616:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9aa:0x0](ffff920a47eb1148) 00000080:00200000:3.0:1788572250.108668:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108675:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108681:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ab:0x0](ffff920a47eb21c8) 00000080:00200000:3.0:1788572250.108695:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ac:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108698:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ac:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108702:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ac:0x0](ffff920a47eb4b08) 00000080:00200000:3.0:1788572250.108715:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ad:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108719:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ad:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108723:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ad:0x0](ffff920a47eb1988) 00000080:00200000:3.0:1788572250.108736:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ae:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108740:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ae:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108744:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ae:0x0](ffff920a47e97448) 00000080:00200000:3.0:1788572250.108758:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9af:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108762:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9af:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9af:0x0](ffff920a47e93248) 00000080:00200000:3.0:1788572250.108778:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108782:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b0:0x0](ffff920a47e95348) 00000080:00200000:3.0:1788572250.108799:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108803:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108807:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b1:0x0](ffff920a47e96c08) 00000080:00200000:3.0:1788572250.108818:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108822:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108826:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b2:0x0](ffff920a47eb63c8) 00000080:00200000:3.0:1788572250.108838:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108842:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b3:0x0](ffff920a47eb00c8) 00000080:00200000:3.0:1788572250.108858:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108862:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108865:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b4:0x0](ffff920a47eb7448) 00000080:00200000:3.0:1788572250.108876:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108880:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b5:0x0](ffff920a47e95b88) 00000080:00200000:3.0:1788572250.108895:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108899:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108902:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b6:0x0](ffff920a47e92a08) 00000080:00200000:3.0:1788572250.108913:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108917:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b7:0x0](ffff920a47e942c8) 00000080:00200000:3.0:1788572250.108946:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108951:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108954:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b8:0x0](ffff920a47e90908) 00000080:00200000:3.0:1788572250.108969:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9b9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108972:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9b9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9b9:0x0](ffff920a487dba88) 00000080:00200000:3.0:1788572250.108989:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ba:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.108993:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ba:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.108997:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ba:0x0](ffff920a487d9148) 00000080:00200000:3.0:1788572250.109016:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9bb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109020:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9bb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109023:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9bb:0x0](ffff920a47eb3248) 00000080:00200000:3.0:1788572250.109037:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9bc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109040:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9bc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9bc:0x0](ffff920a47eb5348) 00000080:00200000:3.0:1788572250.109057:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9bd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109073:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9bd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9bd:0x0](ffff920a47eab248) 00000080:00200000:3.0:1788572250.109092:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9be:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109096:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9be:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109099:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9be:0x0](ffff920a487da1c8) 00000080:00200000:3.0:1788572250.109114:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9bf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109240:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9bf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109246:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9bf:0x0](ffff920a487d9988) 00000080:00200000:3.0:1788572250.109264:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109268:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109271:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c0:0x0](ffff920a487dcb08) 00000080:00200000:3.0:1788572250.109282:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109286:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109289:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c1:0x0](ffff920a487de3c8) 00000080:00200000:3.0:1788572250.109300:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109303:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c2:0x0](ffff920a487d80c8) 00000080:00200000:3.0:1788572250.109318:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109322:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c3:0x0](ffff920a487df448) 00000080:00200000:3.0:1788572250.109336:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109340:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c4:0x0](ffff920a47ead348) 00000080:00200000:3.0:1788572250.109354:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109358:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c5:0x0](ffff920a47eaec08) 00000080:00200000:3.0:1788572250.109373:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109376:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c6:0x0](ffff920a487db248) 00000080:00200000:3.0:1788572250.109467:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109471:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109474:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c7:0x0](ffff920a487dd348) 00000080:00200000:3.0:1788572250.109487:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109490:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c8:0x0](ffff920a487dec08) 00000080:00200000:3.0:1788572250.109504:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9c9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109507:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9c9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9c9:0x0](ffff920a487ddb88) 00000080:00200000:3.0:1788572250.109521:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9ca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109524:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9ca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109527:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9ca:0x0](ffff920a487daa08) 00000080:00200000:3.0:1788572250.109538:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x2000182d1:0x9cb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572250.109542:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x2000182d1:0x9cb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000182d1:0x9cb:0x0](ffff920a487dc2c8) 00000080:00200000:3.0:1788572250.109556:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e1:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109566:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109616:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e1:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109620:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e1:0x1:0x0](ffff920a48633248) 00000080:00200000:3.0:1788572250.109658:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109663:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109670:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109674:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1:0x0](ffff920a487d8908) 00000080:00200000:3.0:1788572250.109694:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x61:0x0](ffff920a48794b08) 00000080:00200000:3.0:1788572250.109703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xa30:0x0](ffff920a47200908) 00000080:00200000:3.0:1788572250.109713:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109717:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109723:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109727:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe1c:0x0](ffff920a4707aa08) 00000080:00200000:3.0:1788572250.109745:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe1e:0x0](ffff920a4705aa08) 00000080:00200000:3.0:1788572250.109756:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109761:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109767:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe45:0x0](ffff920a47097448) 00000080:00200000:3.0:1788572250.109792:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109796:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109801:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe83:0x0](ffff920a4705cb08) 00000080:00200000:3.0:1788572250.109824:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400007ed:0x1:0x0](ffff920a47091988) 00000080:00200000:3.0:1788572250.109831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe98:0x0](ffff920a47094b08) 00000080:00200000:3.0:1788572250.109839:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe99:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109843:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109848:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe99:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109852:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe99:0x0](ffff920a4709db88) 00000080:00200000:3.0:1788572250.109869:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109873:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109878:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9a:0x0](ffff920a4709aa08) 00000080:00200000:3.0:1788572250.109901:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109906:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109913:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9b:0x0](ffff920a470580c8) 00000080:00200000:3.0:1788572250.109947:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109952:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109959:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109962:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9c:0x0](ffff920a4705f448) 00000080:00200000:3.0:1788572250.109981:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.109985:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.109990:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.109994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9d:0x0](ffff920a47093a88) 00000080:00200000:3.0:1788572250.110013:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110017:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110022:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9e:0x0](ffff920a4705b248) 00000080:00200000:3.0:1788572250.110042:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xe9f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110046:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110051:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xe9f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110055:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xe9f:0x0](ffff920a4705e3c8) 00000080:00200000:3.0:1788572250.110242:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110248:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110256:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110260:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea0:0x0](ffff920a470963c8) 00000080:00200000:3.0:1788572250.110282:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110286:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110291:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea1:0x0](ffff920a470900c8) 00000080:00200000:3.0:1788572250.110311:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110315:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110319:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110323:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea2:0x0](ffff920a4709c2c8) 00000080:00200000:3.0:1788572250.110341:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110345:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110350:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110353:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea3:0x0](ffff920a47098908) 00000080:00200000:3.0:1788572250.110368:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110372:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110377:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea4:0x0](ffff920a4709ba88) 00000080:00200000:3.0:1788572250.110475:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110480:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110487:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110490:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea5:0x0](ffff920a47095348) 00000080:00200000:3.0:1788572250.110510:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110515:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110520:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110524:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea6:0x0](ffff920a470921c8) 00000080:00200000:3.0:1788572250.110541:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110545:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110550:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea7:0x0](ffff920a47092a08) 00000080:00200000:3.0:1788572250.110569:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110573:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110578:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110582:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea8:0x0](ffff920a47096c08) 00000080:00200000:3.0:1788572250.110664:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xea9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110670:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110678:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xea9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110682:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xea9:0x0](ffff920a47043248) 00000080:00200000:3.0:1788572250.110702:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0xeab:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.110706:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.110711:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0xeab:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.110715:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xeab:0x0](ffff920a47047448) 00000080:00200000:3.0:1788572250.110736:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xeae:0x0](ffff920a4705d348) 00000080:00200000:3.0:1788572250.110746:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024620:0x3:0x0](ffff920a47046c08) 00000080:00200000:3.0:1788572250.110753:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400007ee:0x3:0x0](ffff920a47045b88) 00000080:00200000:3.0:1788572250.110764:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0xeaf:0x0](ffff920a47095b88) 00000080:00200000:3.0:1788572250.110769:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024620:0x4:0x0](ffff920a47091148) 00000080:00200000:3.0:1788572250.110776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400007ee:0x4:0x0](ffff920a47093248) 00000080:00200000:3.0:1788572250.110787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000400:0x4:0x0](ffff920a470442c8) 00000080:00200000:3.0:1788572250.110794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200000401:0x4:0x0](ffff920a47040908) 00000080:00200000:3.0:1788572250.110801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024620:0x5:0x0](ffff920a47041148) 00000080:00200000:3.0:1788572250.110808:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2400007ee:0x5:0x0](ffff920a470421c8) 00000080:00200000:3.0:1788572250.110815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3:0x0](ffff920a470a3a88) 00000080:00200000:3.0:1788572250.110822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3:0x0](ffff920a470a21c8) 00000080:00200000:3.0:1788572250.110829:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4:0x0](ffff920a472cd348) 00000080:00200000:3.0:1788572250.110836:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4:0x0](ffff920a472cb248) 00000080:00200000:3.0:1788572250.110842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3:0x0](ffff920a472cdb88) 00000080:00200000:3.0:1788572250.110849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3:0x0](ffff920a472c9148) 00000080:00200000:3.0:1788572250.110856:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4:0x0](ffff920a472ce3c8) 00000080:00200000:3.0:1788572250.110862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4:0x0](ffff920a472cec08) 00000080:00200000:3.0:1788572250.110869:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x5:0x0](ffff920a48632a08) 00000080:00200000:3.0:1788572250.110876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x5:0x0](ffff920a48637448) 00000080:00200000:3.0:1788572250.110884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x6:0x0](ffff920a48651988) 00000080:00200000:3.0:1788572250.110891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x6:0x0](ffff920a48655b88) 00000080:00200000:3.0:1788572250.110899:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5:0x0](ffff920a485c21c8) 00000080:00200000:3.0:1788572250.110906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5:0x0](ffff920a485c5b88) 00000080:00200000:3.0:1788572250.110913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x7:0x0](ffff920a4707f448) 00000080:00200000:3.0:1788572250.110921:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x7:0x0](ffff920a47078908) 00000080:00200000:3.0:1788572250.110928:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6:0x0](ffff920a470780c8) 00000080:00200000:3.0:1788572250.110946:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6:0x0](ffff920a4707a1c8) 00000080:00200000:3.0:1788572250.110952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7:0x0](ffff920a487963c8) 00000080:00200000:3.0:1788572250.110960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7:0x0](ffff920a47e83248) 00000080:00200000:3.0:1788572250.110968:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8:0x0](ffff920a472700c8) 00000080:00200000:3.0:1788572250.110975:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8:0x0](ffff920a472763c8) 00000080:00200000:3.0:1788572250.110982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x8:0x0](ffff920a485c6c08) 00000080:00200000:3.0:1788572250.110988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x8:0x0](ffff920a485c5348) 00000080:00200000:3.0:1788572250.110995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x9:0x0](ffff920a47e84b08) 00000080:00200000:3.0:1788572250.111002:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x9:0x0](ffff920a47e81988) 00000080:00200000:3.0:1788572250.111009:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9:0x0](ffff920a485721c8) 00000080:00200000:3.0:1788572250.111017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9:0x0](ffff920a48577448) 00000080:00200000:3.0:1788572250.111025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xa:0x0](ffff920a48571148) 00000080:00200000:3.0:1788572250.111031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xa:0x0](ffff920a48571988) 00000080:00200000:3.0:1788572250.111037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa:0x0](ffff920a47e86c08) 00000080:00200000:3.0:1788572250.111044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa:0x0](ffff920a47e85b88) 00000080:00200000:3.0:1788572250.111051:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb:0x0](ffff920a47041988) 00000080:00200000:3.0:1788572250.111057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb:0x0](ffff920a47045348) 00000080:00200000:3.0:1788572250.111216:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xb:0x0](ffff920a470463c8) 00000080:00200000:3.0:1788572250.111226:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xb:0x0](ffff920a47044b08) 00000080:00200000:3.0:1788572250.111232:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xc:0x0](ffff920a470a8908) 00000080:00200000:3.0:1788572250.111239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xc:0x0](ffff920a470ac2c8) 00000080:00200000:3.0:1788572250.111246:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xd:0x0](ffff920a4715e3c8) 00000080:00200000:3.0:1788572250.111253:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xd:0x0](ffff920a4715cb08) 00000080:00200000:3.0:1788572250.111261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xe:0x0](ffff920a4715a1c8) 00000080:00200000:3.0:1788572250.111268:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xe:0x0](ffff920a47159148) 00000080:00200000:3.0:1788572250.111275:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0xf:0x0](ffff920a4719aa08) 00000080:00200000:3.0:1788572250.111282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0xf:0x0](ffff920a4719db88) 00000080:00200000:3.0:1788572250.111290:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc:0x0](ffff920a47007448) 00000080:00200000:3.0:1788572250.111297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc:0x0](ffff920a470000c8) 00000080:00200000:3.0:1788572250.111303:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x10:0x0](ffff920a4700ba88) 00000080:00200000:3.0:1788572250.111310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x10:0x0](ffff920a47008908) 00000080:00200000:3.0:1788572250.111318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd:0x0](ffff920a4700db88) 00000080:00200000:3.0:1788572250.111324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd:0x0](ffff920a4700ec08) 00000080:00200000:3.0:1788572250.111330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe:0x0](ffff920a4700b248) 00000080:00200000:3.0:1788572250.111337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe:0x0](ffff920a4700f448) 00000080:00200000:3.0:1788572250.111343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x11:0x0](ffff920a4700cb08) 00000080:00200000:3.0:1788572250.111350:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x11:0x0](ffff920a47009988) 00000080:00200000:3.0:1788572250.111357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf:0x0](ffff920a47e33248) 00000080:00200000:3.0:1788572250.111363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf:0x0](ffff920a47e35348) 00000080:00200000:3.0:1788572250.111370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x12:0x0](ffff920a487ecb08) 00000080:00200000:3.0:1788572250.111377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x12:0x0](ffff920a487e9988) 00000080:00200000:3.0:1788572250.111447:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x13:0x0](ffff920a487e21c8) 00000080:00200000:3.0:1788572250.111455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x13:0x0](ffff920a487e1148) 00000080:00200000:3.0:1788572250.111462:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x14:0x0](ffff920a487e0908) 00000080:00200000:3.0:1788572250.111469:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x14:0x0](ffff920a487e42c8) 00000080:00200000:3.0:1788572250.111476:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x15:0x0](ffff920a487e7448) 00000080:00200000:3.0:1788572250.111484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x15:0x0](ffff920a487e00c8) 00000080:00200000:3.0:1788572250.111490:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x10:0x0](ffff920a487ef448) 00000080:00200000:3.0:1788572250.111497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x10:0x0](ffff920a487e80c8) 00000080:00200000:3.0:1788572250.111503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x16:0x0](ffff920a48723248) 00000080:00200000:3.0:1788572250.111510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x16:0x0](ffff920a48727448) 00000080:00200000:3.0:1788572250.111517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x17:0x0](ffff920a4872a1c8) 00000080:00200000:3.0:1788572250.111525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x17:0x0](ffff920a48729148) 00000080:00200000:3.0:1788572250.111533:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x11:0x0](ffff920a47ec5348) 00000080:00200000:3.0:1788572250.111540:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x11:0x0](ffff920a47ec3248) 00000080:00200000:3.0:1788572250.111546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x12:0x0](ffff920a47ec0908) 00000080:00200000:3.0:1788572250.111553:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x12:0x0](ffff920a47ec42c8) 00000080:00200000:3.0:1788572250.111560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x18:0x0](ffff920a47e30908) 00000080:00200000:3.0:1788572250.111566:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x18:0x0](ffff920a47e33a88) 00000080:00200000:3.0:1788572250.111573:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x13:0x0](ffff920a470642c8) 00000080:00200000:3.0:1788572250.111579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x13:0x0](ffff920a47062a08) 00000080:00200000:3.0:1788572250.111666:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x14:0x0](ffff920a47ecc2c8) 00000080:00200000:3.0:1788572250.111676:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x14:0x0](ffff920a47ecaa08) 00000080:00200000:3.0:1788572250.111683:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x15:0x0](ffff920a47ecec08) 00000080:00200000:3.0:1788572250.111689:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x15:0x0](ffff920a47ecd348) 00000080:00200000:3.0:1788572250.111695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x19:0x0](ffff920a47065b88) 00000080:00200000:3.0:1788572250.111703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x19:0x0](ffff920a47066c08) 00000080:00200000:3.0:1788572250.111709:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x16:0x0](ffff920a47067448) 00000080:00200000:3.0:1788572250.111716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x16:0x0](ffff920a470600c8) 00000080:00200000:3.0:1788572250.111722:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1a:0x0](ffff920a47ec80c8) 00000080:00200000:3.0:1788572250.111729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1a:0x0](ffff920a47ece3c8) 00000080:00200000:3.0:1788572250.111736:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1b:0x0](ffff920a47ec9988) 00000080:00200000:3.0:1788572250.111742:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1b:0x0](ffff920a47eca1c8) 00000080:00200000:3.0:1788572250.111749:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x17:0x0](ffff920a47ecba88) 00000080:00200000:3.0:1788572250.111756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x17:0x0](ffff920a47ec8908) 00000080:00200000:3.0:1788572250.111762:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x18:0x0](ffff920a47f71988) 00000080:00200000:3.0:1788572250.111770:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x18:0x0](ffff920a47f721c8) 00000080:00200000:3.0:1788572250.111776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1c:0x0](ffff920a47f70908) 00000080:00200000:3.0:1788572250.111783:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1c:0x0](ffff920a47f742c8) 00000080:00200000:3.0:1788572250.111789:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1d:0x0](ffff920a47f75b88) 00000080:00200000:3.0:1788572250.111797:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1d:0x0](ffff920a47f76c08) 00000080:00200000:3.0:1788572250.111804:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1e:0x0](ffff920a47f7ec08) 00000080:00200000:3.0:1788572250.111816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1e:0x0](ffff920a47f7d348) 00000080:00200000:3.0:1788572250.111823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x1f:0x0](ffff920a47f780c8) 00000080:00200000:3.0:1788572250.111831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x1f:0x0](ffff920a47f7e3c8) 00000080:00200000:3.0:1788572250.111838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x20:0x0](ffff920a47f7a1c8) 00000080:00200000:3.0:1788572250.111846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x20:0x0](ffff920a47f79148) 00000080:00200000:3.0:1788572250.111854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x21:0x0](ffff920a47f77448) 00000080:00200000:3.0:1788572250.111863:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x21:0x0](ffff920a47f700c8) 00000080:00200000:3.0:1788572250.111871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x19:0x0](ffff920a47fdc2c8) 00000080:00200000:3.0:1788572250.111878:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x19:0x0](ffff920a47fdaa08) 00000080:00200000:3.0:1788572250.111884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1a:0x0](ffff920a47019148) 00000080:00200000:3.0:1788572250.111892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1a:0x0](ffff920a4701a1c8) 00000080:00200000:3.0:1788572250.111900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x22:0x0](ffff920a47fd1988) 00000080:00200000:3.0:1788572250.111909:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x22:0x0](ffff920a47fd21c8) 00000080:00200000:3.0:1788572250.111917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x23:0x0](ffff920a47fdd348) 00000080:00200000:3.0:1788572250.111924:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x23:0x0](ffff920a47fdb248) 00000080:00200000:3.0:1788572250.111942:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1b:0x0](ffff920a47fd80c8) 00000080:00200000:3.0:1788572250.111951:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1b:0x0](ffff920a47fde3c8) 00000080:00200000:3.0:1788572250.111958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1c:0x0](ffff920a47fd3a88) 00000080:00200000:3.0:1788572250.111965:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1c:0x0](ffff920a47fd0908) 00000080:00200000:3.0:1788572250.111972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1d:0x0](ffff920a47fd2a08) 00000080:00200000:3.0:1788572250.111978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1d:0x0](ffff920a47fd5b88) 00000080:00200000:3.0:1788572250.111985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x24:0x0](ffff920a47fd5348) 00000080:00200000:3.0:1788572250.111992:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x24:0x0](ffff920a47fd3248) 00000080:00200000:3.0:1788572250.111999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1e:0x0](ffff920a47fd9988) 00000080:00200000:3.0:1788572250.112006:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1e:0x0](ffff920a47fda1c8) 00000080:00200000:3.0:1788572250.112014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x25:0x0](ffff920a470180c8) 00000080:00200000:3.0:1788572250.112022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x25:0x0](ffff920a4701f448) 00000080:00200000:3.0:1788572250.112029:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x26:0x0](ffff920a472121c8) 00000080:00200000:3.0:1788572250.112037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x26:0x0](ffff920a47211148) 00000080:00200000:3.0:1788572250.112044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x1f:0x0](ffff920a4721b248) 00000080:00200000:3.0:1788572250.112052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x1f:0x0](ffff920a4721f448) 00000080:00200000:3.0:1788572250.112059:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x27:0x0](ffff920a47221988) 00000080:00200000:3.0:1788572250.112267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x27:0x0](ffff920a472221c8) 00000080:00200000:3.0:1788572250.112276:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x20:0x0](ffff920a47223a88) 00000080:00200000:3.0:1788572250.112283:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x20:0x0](ffff920a47220908) 00000080:00200000:3.0:1788572250.112292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x21:0x0](ffff920a47219148) 00000080:00200000:3.0:1788572250.112301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x21:0x0](ffff920a47229988) 00000080:00200000:3.0:1788572250.112309:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x22:0x0](ffff920a47229148) 00000080:00200000:3.0:1788572250.112319:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x22:0x0](ffff920a4722ba88) 00000080:00200000:3.0:1788572250.112327:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x23:0x0](ffff920a472042c8) 00000080:00200000:3.0:1788572250.112337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x23:0x0](ffff920a47202a08) 00000080:00200000:3.0:1788572250.112345:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x24:0x0](ffff920a4722aa08) 00000080:00200000:3.0:1788572250.112354:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x24:0x0](ffff920a4722db88) 00000080:00200000:3.0:1788572250.112363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x28:0x0](ffff920a4722f448) 00000080:00200000:3.0:1788572250.112372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x28:0x0](ffff920a472280c8) 00000080:00200000:3.0:1788572250.112449:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x29:0x0](ffff920a4722cb08) 00000080:00200000:3.0:1788572250.112460:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x29:0x0](ffff920a472300c8) 00000080:00200000:3.0:1788572250.112468:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2a:0x0](ffff920a47234b08) 00000080:00200000:3.0:1788572250.112477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2a:0x0](ffff920a47231988) 00000080:00200000:3.0:1788572250.112484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2b:0x0](ffff920a47231148) 00000080:00200000:3.0:1788572250.112492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2b:0x0](ffff920a47233a88) 00000080:00200000:3.0:1788572250.112499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x25:0x0](ffff920a47226c08) 00000080:00200000:3.0:1788572250.112508:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x25:0x0](ffff920a47225348) 00000080:00200000:3.0:1788572250.112515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x26:0x0](ffff920a4723e3c8) 00000080:00200000:3.0:1788572250.112522:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x26:0x0](ffff920a4723cb08) 00000080:00200000:3.0:1788572250.112529:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x27:0x0](ffff920a4723a1c8) 00000080:00200000:3.0:1788572250.112538:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x27:0x0](ffff920a47239148) 00000080:00200000:3.0:1788572250.112546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2c:0x0](ffff920a47233248) 00000080:00200000:3.0:1788572250.112554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2c:0x0](ffff920a47237448) 00000080:00200000:3.0:1788572250.112562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x28:0x0](ffff920a47165348) 00000080:00200000:3.0:1788572250.112571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x28:0x0](ffff920a47163248) 00000080:00200000:3.0:1788572250.112577:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x29:0x0](ffff920a4723db88) 00000080:00200000:3.0:1788572250.112709:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x29:0x0](ffff920a4723ec08) 00000080:00200000:3.0:1788572250.112721:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2a:0x0](ffff920a47161988) 00000080:00200000:3.0:1788572250.112730:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2a:0x0](ffff920a471621c8) 00000080:00200000:3.0:1788572250.112738:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2b:0x0](ffff920a471642c8) 00000080:00200000:3.0:1788572250.112747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2b:0x0](ffff920a47162a08) 00000080:00200000:3.0:1788572250.112756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2c:0x0](ffff920a4704b248) 00000080:00200000:3.0:1788572250.112767:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2c:0x0](ffff920a4704d348) 00000080:00200000:3.0:1788572250.112776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2d:0x0](ffff920a472380c8) 00000080:00200000:3.0:1788572250.112785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2d:0x0](ffff920a47176c08) 00000080:00200000:3.0:1788572250.112793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2e:0x0](ffff920a4716b248) 00000080:00200000:3.0:1788572250.112802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2e:0x0](ffff920a4716f448) 00000080:00200000:3.0:1788572250.112810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x2f:0x0](ffff920a47177448) 00000080:00200000:3.0:1788572250.112818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x2f:0x0](ffff920a471700c8) 00000080:00200000:3.0:1788572250.112826:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2d:0x0](ffff920a471721c8) 00000080:00200000:3.0:1788572250.112835:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2d:0x0](ffff920a47171148) 00000080:00200000:3.0:1788572250.112842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2e:0x0](ffff920a4717c2c8) 00000080:00200000:3.0:1788572250.112850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2e:0x0](ffff920a4717aa08) 00000080:00200000:3.0:1788572250.112857:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x2f:0x0](ffff920a4717ec08) 00000080:00200000:3.0:1788572250.112865:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x2f:0x0](ffff920a4717d348) 00000080:00200000:3.0:1788572250.112873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x30:0x0](ffff920a4717f448) 00000080:00200000:3.0:1788572250.112881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x30:0x0](ffff920a471780c8) 00000080:00200000:3.0:1788572250.112890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x30:0x0](ffff920a4876c2c8) 00000080:00200000:3.0:1788572250.112898:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x30:0x0](ffff920a48777448) 00000080:00200000:3.0:1788572250.112906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x31:0x0](ffff920a487621c8) 00000080:00200000:3.0:1788572250.112914:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x31:0x0](ffff920a48761148) 00000080:00200000:3.0:1788572250.112922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x31:0x0](ffff920a487763c8) 00000080:00200000:3.0:1788572250.112945:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x31:0x0](ffff920a48774b08) 00000080:00200000:3.0:1788572250.112955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x32:0x0](ffff920a4704a1c8) 00000080:00200000:3.0:1788572250.112964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x32:0x0](ffff920a47049988) 00000080:00200000:3.0:1788572250.112972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x32:0x0](ffff920a487780c8) 00000080:00200000:3.0:1788572250.112981:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x32:0x0](ffff920a4877e3c8) 00000080:00200000:3.0:1788572250.112990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x33:0x0](ffff920a48779988) 00000080:00200000:3.0:1788572250.112999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x33:0x0](ffff920a4877a1c8) 00000080:00200000:3.0:1788572250.113007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x34:0x0](ffff920a48773a88) 00000080:00200000:3.0:1788572250.113014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x34:0x0](ffff920a48770908) 00000080:00200000:3.0:1788572250.113022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x33:0x0](ffff920a48775348) 00000080:00200000:3.0:1788572250.113030:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x33:0x0](ffff920a48773248) 00000080:00200000:3.0:1788572250.113038:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x35:0x0](ffff920a48778908) 00000080:00200000:3.0:1788572250.113046:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x35:0x0](ffff920a4877c2c8) 00000080:00200000:3.0:1788572250.113056:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x36:0x0](ffff920a47204b08) 00000080:00200000:3.0:1788572250.113192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x36:0x0](ffff920a47207448) 00000080:00200000:3.0:1788572250.113205:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x37:0x0](ffff920a471063c8) 00000080:00200000:3.0:1788572250.113213:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x37:0x0](ffff920a47104b08) 00000080:00200000:3.0:1788572250.113221:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x38:0x0](ffff920a47101148) 00000080:00200000:3.0:1788572250.113229:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x38:0x0](ffff920a47103a88) 00000080:00200000:3.0:1788572250.113239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x34:0x0](ffff920a471042c8) 00000080:00200000:3.0:1788572250.113247:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x34:0x0](ffff920a47102a08) 00000080:00200000:3.0:1788572250.113256:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x35:0x0](ffff920a4710b248) 00000080:00200000:3.0:1788572250.113266:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x35:0x0](ffff920a4710f448) 00000080:00200000:3.0:1788572250.113274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x36:0x0](ffff920a47113248) 00000080:00200000:3.0:1788572250.113282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x36:0x0](ffff920a47117448) 00000080:00200000:3.0:1788572250.113291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x39:0x0](ffff920a4710e3c8) 00000080:00200000:3.0:1788572250.113301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x39:0x0](ffff920a4710cb08) 00000080:00200000:3.0:1788572250.113310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3a:0x0](ffff920a47114b08) 00000080:00200000:3.0:1788572250.113318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3a:0x0](ffff920a47111988) 00000080:00200000:3.0:1788572250.113327:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3b:0x0](ffff920a47109148) 00000080:00200000:3.0:1788572250.113335:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3b:0x0](ffff920a4710ba88) 00000080:00200000:3.0:1788572250.113343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3c:0x0](ffff920a47115348) 00000080:00200000:3.0:1788572250.113351:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3c:0x0](ffff920a47021148) 00000080:00200000:3.0:1788572250.113362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3d:0x0](ffff920a470242c8) 00000080:00200000:3.0:1788572250.113372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3d:0x0](ffff920a47022a08) 00000080:00200000:3.0:1788572250.113454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x37:0x0](ffff920a47119988) 00000080:00200000:3.0:1788572250.113466:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x37:0x0](ffff920a4711a1c8) 00000080:00200000:3.0:1788572250.113475:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x38:0x0](ffff920a47119148) 00000080:00200000:3.0:1788572250.113483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x38:0x0](ffff920a4711ba88) 00000080:00200000:3.0:1788572250.113491:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x39:0x0](ffff920a4711c2c8) 00000080:00200000:3.0:1788572250.113499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x39:0x0](ffff920a4711aa08) 00000080:00200000:3.0:1788572250.113507:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3a:0x0](ffff920a4711ec08) 00000080:00200000:3.0:1788572250.113515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3a:0x0](ffff920a4711d348) 00000080:00200000:3.0:1788572250.113523:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3b:0x0](ffff920a4720d348) 00000080:00200000:3.0:1788572250.113531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3b:0x0](ffff920a4720b248) 00000080:00200000:3.0:1788572250.113538:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3e:0x0](ffff920a472080c8) 00000080:00200000:3.0:1788572250.113546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3e:0x0](ffff920a4720db88) 00000080:00200000:3.0:1788572250.113554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x3f:0x0](ffff920a47023248) 00000080:00200000:3.0:1788572250.113562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x3f:0x0](ffff920a47027448) 00000080:00200000:3.0:1788572250.113571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x40:0x0](ffff920a47029148) 00000080:00200000:3.0:1788572250.113579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x40:0x0](ffff920a4702ba88) 00000080:00200000:3.0:1788572250.113707:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x41:0x0](ffff920a470263c8) 00000080:00200000:3.0:1788572250.113720:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x41:0x0](ffff920a47024b08) 00000080:00200000:3.0:1788572250.113729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x42:0x0](ffff920a470221c8) 00000080:00200000:3.0:1788572250.113736:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x42:0x0](ffff920a47031988) 00000080:00200000:3.0:1788572250.113744:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x43:0x0](ffff920a47031148) 00000080:00200000:3.0:1788572250.113752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x43:0x0](ffff920a47033a88) 00000080:00200000:3.0:1788572250.113760:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x44:0x0](ffff920a4702c2c8) 00000080:00200000:3.0:1788572250.113769:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x44:0x0](ffff920a4702aa08) 00000080:00200000:3.0:1788572250.113777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3c:0x0](ffff920a4702cb08) 00000080:00200000:3.0:1788572250.113785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3c:0x0](ffff920a47029988) 00000080:00200000:3.0:1788572250.113793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3d:0x0](ffff920a47ee3a88) 00000080:00200000:3.0:1788572250.113802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3d:0x0](ffff920a47ee0908) 00000080:00200000:3.0:1788572250.113810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x45:0x0](ffff920a4703cb08) 00000080:00200000:3.0:1788572250.113818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x45:0x0](ffff920a47039988) 00000080:00200000:3.0:1788572250.113827:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x46:0x0](ffff920a47039148) 00000080:00200000:3.0:1788572250.113835:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x46:0x0](ffff920a4703ba88) 00000080:00200000:3.0:1788572250.113843:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x47:0x0](ffff920a47ee5348) 00000080:00200000:3.0:1788572250.113851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x47:0x0](ffff920a47ee3248) 00000080:00200000:3.0:1788572250.113858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3e:0x0](ffff920a47eedb88) 00000080:00200000:3.0:1788572250.113866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3e:0x0](ffff920a47eeec08) 00000080:00200000:3.0:1788572250.113874:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x48:0x0](ffff920a47eeb248) 00000080:00200000:3.0:1788572250.113882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x48:0x0](ffff920a47eef448) 00000080:00200000:3.0:1788572250.113890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x3f:0x0](ffff920a47ef1148) 00000080:00200000:3.0:1788572250.113898:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x3f:0x0](ffff920a47ef3a88) 00000080:00200000:3.0:1788572250.113905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x40:0x0](ffff920a47eecb08) 00000080:00200000:3.0:1788572250.113914:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x40:0x0](ffff920a47ef80c8) 00000080:00200000:3.0:1788572250.113921:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x49:0x0](ffff920a47efcb08) 00000080:00200000:3.0:1788572250.113941:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x49:0x0](ffff920a47ef9988) 00000080:00200000:3.0:1788572250.113950:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4a:0x0](ffff920a47ef42c8) 00000080:00200000:3.0:1788572250.113960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4a:0x0](ffff920a47ef2a08) 00000080:00200000:3.0:1788572250.113967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4b:0x0](ffff920a47ef5348) 00000080:00200000:3.0:1788572250.113976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4b:0x0](ffff920a47ef3248) 00000080:00200000:3.0:1788572250.113985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x41:0x0](ffff920a47ef9148) 00000080:00200000:3.0:1788572250.113993:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x41:0x0](ffff920a47efba88) 00000080:00200000:3.0:1788572250.114001:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x42:0x0](ffff920a487a8908) 00000080:00200000:3.0:1788572250.114009:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x42:0x0](ffff920a487ac2c8) 00000080:00200000:3.0:1788572250.114016:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4c:0x0](ffff920a487a6c08) 00000080:00200000:3.0:1788572250.114024:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4c:0x0](ffff920a487a5348) 00000080:00200000:3.0:1788572250.114032:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x43:0x0](ffff920a487a7448) 00000080:00200000:3.0:1788572250.114040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x43:0x0](ffff920a487a00c8) 00000080:00200000:3.0:1788572250.114049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4d:0x0](ffff920a487adb88) 00000080:00200000:3.0:1788572250.114058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4d:0x0](ffff920a487aec08) 00000080:00200000:3.0:1788572250.114081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4e:0x0](ffff920a487a4b08) 00000080:00200000:3.0:1788572250.114092:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4e:0x0](ffff920a487b3248) 00000080:00200000:3.0:1788572250.114100:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x4f:0x0](ffff920a487b00c8) 00000080:00200000:3.0:1788572250.114110:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x4f:0x0](ffff920a487b63c8) 00000080:00200000:3.0:1788572250.114246:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x50:0x0](ffff920a487ab248) 00000080:00200000:3.0:1788572250.114261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x50:0x0](ffff920a487af448) 00000080:00200000:3.0:1788572250.114270:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x51:0x0](ffff920a487b1988) 00000080:00200000:3.0:1788572250.114280:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x51:0x0](ffff920a487b21c8) 00000080:00200000:3.0:1788572250.114289:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x52:0x0](ffff920a487b3a88) 00000080:00200000:3.0:1788572250.114297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x52:0x0](ffff920a487b0908) 00000080:00200000:3.0:1788572250.114306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x53:0x0](ffff920a487ae3c8) 00000080:00200000:3.0:1788572250.114315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x53:0x0](ffff920a487acb08) 00000080:00200000:3.0:1788572250.114324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x54:0x0](ffff920a487b5b88) 00000080:00200000:3.0:1788572250.114332:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x54:0x0](ffff920a487b6c08) 00000080:00200000:3.0:1788572250.114341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x44:0x0](ffff920a487bcb08) 00000080:00200000:3.0:1788572250.114348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x44:0x0](ffff920a487b9988) 00000080:00200000:3.0:1788572250.114357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x45:0x0](ffff920a487baa08) 00000080:00200000:3.0:1788572250.114366:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x45:0x0](ffff920a487bdb88) 00000080:00200000:3.0:1788572250.114374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x46:0x0](ffff920a47c4ba88) 00000080:00200000:3.0:1788572250.114458:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x46:0x0](ffff920a47c48908) 00000080:00200000:3.0:1788572250.114470:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x55:0x0](ffff920a4720a1c8) 00000080:00200000:3.0:1788572250.114479:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x55:0x0](ffff920a47209148) 00000080:00200000:3.0:1788572250.114486:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x56:0x0](ffff920a47c4d348) 00000080:00200000:3.0:1788572250.114495:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x56:0x0](ffff920a47c4b248) 00000080:00200000:3.0:1788572250.114503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x57:0x0](ffff920a472a4b08) 00000080:00200000:3.0:1788572250.114511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x57:0x0](ffff920a472a63c8) 00000080:00200000:3.0:1788572250.114520:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x58:0x0](ffff920a47c421c8) 00000080:00200000:3.0:1788572250.114529:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x58:0x0](ffff920a47c41148) 00000080:00200000:3.0:1788572250.114537:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x47:0x0](ffff920a47c40908) 00000080:00200000:3.0:1788572250.114545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x47:0x0](ffff920a47c442c8) 00000080:00200000:3.0:1788572250.114553:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x48:0x0](ffff920a47c563c8) 00000080:00200000:3.0:1788572250.114561:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x48:0x0](ffff920a47c54b08) 00000080:00200000:3.0:1788572250.114568:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x59:0x0](ffff920a47c521c8) 00000080:00200000:3.0:1788572250.114576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x59:0x0](ffff920a47c51148) 00000080:00200000:3.0:1788572250.114582:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x49:0x0](ffff920a47c49988) 00000080:00200000:3.0:1788572250.114721:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x49:0x0](ffff920a47c4a1c8) 00000080:00200000:3.0:1788572250.114730:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4a:0x0](ffff920a47c59148) 00000080:00200000:3.0:1788572250.114739:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4a:0x0](ffff920a47c5ba88) 00000080:00200000:3.0:1788572250.114747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5a:0x0](ffff920a47c5c2c8) 00000080:00200000:3.0:1788572250.114756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5a:0x0](ffff920a47c5aa08) 00000080:00200000:3.0:1788572250.114765:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5b:0x0](ffff920a47c52a08) 00000080:00200000:3.0:1788572250.114773:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5b:0x0](ffff920a47c55b88) 00000080:00200000:3.0:1788572250.114781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4b:0x0](ffff920a47c53248) 00000080:00200000:3.0:1788572250.114789:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4b:0x0](ffff920a47c57448) 00000080:00200000:3.0:1788572250.114797:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4c:0x0](ffff920a47f44b08) 00000080:00200000:3.0:1788572250.114806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4c:0x0](ffff920a47f41988) 00000080:00200000:3.0:1788572250.114815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5c:0x0](ffff920a47f41148) 00000080:00200000:3.0:1788572250.114823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5c:0x0](ffff920a47f43a88) 00000080:00200000:3.0:1788572250.114832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5d:0x0](ffff920a472a5b88) 00000080:00200000:3.0:1788572250.114842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5d:0x0](ffff920a472a2a08) 00000080:00200000:3.0:1788572250.114850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5e:0x0](ffff920a47f4e3c8) 00000080:00200000:3.0:1788572250.114859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5e:0x0](ffff920a47f4cb08) 00000080:00200000:3.0:1788572250.114868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x5f:0x0](ffff920a47f4a1c8) 00000080:00200000:3.0:1788572250.114877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x5f:0x0](ffff920a47f49148) 00000080:00200000:3.0:1788572250.114885:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x60:0x0](ffff920a47f45348) 00000080:00200000:3.0:1788572250.114893:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x60:0x0](ffff920a47f43248) 00000080:00200000:3.0:1788572250.114902:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4d:0x0](ffff920a47f4c2c8) 00000080:00200000:3.0:1788572250.114910:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4d:0x0](ffff920a47f4aa08) 00000080:00200000:3.0:1788572250.114919:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4e:0x0](ffff920a47f55348) 00000080:00200000:3.0:1788572250.114941:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4e:0x0](ffff920a47f53248) 00000080:00200000:3.0:1788572250.114952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x4f:0x0](ffff920a4865a1c8) 00000080:00200000:3.0:1788572250.114961:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x4f:0x0](ffff920a4865db88) 00000080:00200000:3.0:1788572250.114971:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x50:0x0](ffff920a48658908) 00000080:00200000:3.0:1788572250.114980:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x50:0x0](ffff920a48520908) 00000080:00200000:3.0:1788572250.114991:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x51:0x0](ffff920a47f51988) 00000080:00200000:3.0:1788572250.114999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x51:0x0](ffff920a47f521c8) 00000080:00200000:3.0:1788572250.115008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x52:0x0](ffff920a47f53a88) 00000080:00200000:3.0:1788572250.115020:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x52:0x0](ffff920a47f50908) 00000080:00200000:3.0:1788572250.115029:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x61:0x0](ffff920a47f542c8) 00000080:00200000:3.0:1788572250.115037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x61:0x0](ffff920a47f52a08) 00000080:00200000:3.0:1788572250.115044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x62:0x0](ffff920a47f59148) 00000080:00200000:3.0:1788572250.115053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x62:0x0](ffff920a47f5ba88) 00000080:00200000:3.0:1788572250.115071:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x53:0x0](ffff920a47fe1148) 00000080:00200000:3.0:1788572250.115081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x53:0x0](ffff920a47fe3a88) 00000080:00200000:3.0:1788572250.115089:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x63:0x0](ffff920a47f5aa08) 00000080:00200000:3.0:1788572250.115096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x63:0x0](ffff920a47f5db88) 00000080:00200000:3.0:1788572250.115104:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x64:0x0](ffff920a472a21c8) 00000080:00200000:3.0:1788572250.115114:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x64:0x0](ffff920a472a1988) 00000080:00200000:3.0:1788572250.115250:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x54:0x0](ffff920a47ffcb08) 00000080:00200000:3.0:1788572250.115259:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x54:0x0](ffff920a47ff9988) 00000080:00200000:3.0:1788572250.115267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x65:0x0](ffff920a47ff63c8) 00000080:00200000:3.0:1788572250.115275:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x65:0x0](ffff920a47ff4b08) 00000080:00200000:3.0:1788572250.115283:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x66:0x0](ffff920a47ff9148) 00000080:00200000:3.0:1788572250.115291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x66:0x0](ffff920a47ffba88) 00000080:00200000:3.0:1788572250.115300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x55:0x0](ffff920a471c0908) 00000080:00200000:3.0:1788572250.115308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x55:0x0](ffff920a471c42c8) 00000080:00200000:3.0:1788572250.115316:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x67:0x0](ffff920a471c5b88) 00000080:00200000:3.0:1788572250.115324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x67:0x0](ffff920a471c6c08) 00000080:00200000:3.0:1788572250.115331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x56:0x0](ffff920a485263c8) 00000080:00200000:3.0:1788572250.115340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x56:0x0](ffff920a485200c8) 00000080:00200000:3.0:1788572250.115348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x57:0x0](ffff920a47ffaa08) 00000080:00200000:3.0:1788572250.115356:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x57:0x0](ffff920a47ffdb88) 00000080:00200000:3.0:1788572250.115364:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x58:0x0](ffff920a471c5348) 00000080:00200000:3.0:1788572250.115373:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x58:0x0](ffff920a471c3248) 00000080:00200000:3.0:1788572250.115457:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x59:0x0](ffff920a47ffb248) 00000080:00200000:3.0:1788572250.115469:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x59:0x0](ffff920a47fff448) 00000080:00200000:3.0:1788572250.115477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x5a:0x0](ffff920a471c4b08) 00000080:00200000:3.0:1788572250.115485:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x5a:0x0](ffff920a471c1988) 00000080:00200000:3.0:1788572250.115493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb8:0x5b:0x0](ffff920a471d0908) 00000080:00200000:3.0:1788572250.115502:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df1:0x5b:0x0](ffff920a471d42c8) 00000080:00200000:3.0:1788572250.115510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x68:0x0](ffff920a471d5b88) 00000080:00200000:3.0:1788572250.115519:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x68:0x0](ffff920a471d6c08) 00000080:00200000:3.0:1788572250.115527:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1530:0x0](ffff920a47e81148) 00000080:00200000:3.0:1788572250.115539:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x69:0x0](ffff920a470480c8) 00000080:00200000:3.0:1788572250.115548:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x69:0x0](ffff920a4704e3c8) 00000080:00200000:3.0:1788572250.115556:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3:0x0](ffff920a4704c2c8) 00000080:00200000:3.0:1788572250.115563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3:0x0](ffff920a4704db88) 00000080:00200000:3.0:1788572250.115571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6a:0x0](ffff920a47048908) 00000080:00200000:3.0:1788572250.115578:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6a:0x0](ffff920a4704f448) 00000080:00200000:3.0:1788572250.115810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4:0x0](ffff920a47fe8908) 00000080:00200000:3.0:1788572250.115824:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4:0x0](ffff920a47fea1c8) 00000080:00200000:3.0:1788572250.115832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6b:0x0](ffff920a47049148) 00000080:00200000:3.0:1788572250.115841:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6b:0x0](ffff920a4704aa08) 00000080:00200000:3.0:1788572250.115849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5:0x0](ffff920a47e863c8) 00000080:00200000:3.0:1788572250.115859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5:0x0](ffff920a47e821c8) 00000080:00200000:3.0:1788572250.115867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6c:0x0](ffff920a47c58908) 00000080:00200000:3.0:1788572250.115875:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6c:0x0](ffff920a47c5ec08) 00000080:00200000:3.0:1788572250.115882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6:0x0](ffff920a47e82a08) 00000080:00200000:3.0:1788572250.115890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6:0x0](ffff920a47e80908) 00000080:00200000:3.0:1788572250.115898:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6d:0x0](ffff920a47fef448) 00000080:00200000:3.0:1788572250.115906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6d:0x0](ffff920a47fecb08) 00000080:00200000:3.0:1788572250.115914:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7:0x0](ffff920a47feaa08) 00000080:00200000:3.0:1788572250.115922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7:0x0](ffff920a47fec2c8) 00000080:00200000:3.0:1788572250.115942:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6e:0x0](ffff920a4704ba88) 00000080:00200000:3.0:1788572250.115952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6e:0x0](ffff920a4704cb08) 00000080:00200000:3.0:1788572250.115960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8:0x0](ffff920a47e87448) 00000080:00200000:3.0:1788572250.115969:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8:0x0](ffff920a47e842c8) 00000080:00200000:3.0:1788572250.115978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x6f:0x0](ffff920a470342c8) 00000080:00200000:3.0:1788572250.115986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x6f:0x0](ffff920a47033248) 00000080:00200000:3.0:1788572250.115995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x9:0x0](ffff920a47030908) 00000080:00200000:3.0:1788572250.116003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x9:0x0](ffff920a47032a08) 00000080:00200000:3.0:1788572250.116010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x70:0x0](ffff920a47c5cb08) 00000080:00200000:3.0:1788572250.116018:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x70:0x0](ffff920a47c5e3c8) 00000080:00200000:3.0:1788572250.116026:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xa:0x0](ffff920a47feb248) 00000080:00200000:3.0:1788572250.116034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xa:0x0](ffff920a47feec08) 00000080:00200000:3.0:1788572250.116042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x71:0x0](ffff920a48651148) 00000080:00200000:3.0:1788572250.116051:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x71:0x0](ffff920a48654b08) 00000080:00200000:3.0:1788572250.116059:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xb:0x0](ffff920a48655348) 00000080:00200000:3.0:1788572250.116196:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xb:0x0](ffff920a48650908) 00000080:00200000:3.0:1788572250.116204:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x72:0x0](ffff920a47e85348) 00000080:00200000:3.0:1788572250.116212:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x72:0x0](ffff920a4704ec08) 00000080:00200000:3.0:1788572250.116220:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xc:0x0](ffff920a48763a88) 00000080:00200000:3.0:1788572250.116228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xc:0x0](ffff920a48760908) 00000080:00200000:3.0:1788572250.116236:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x73:0x0](ffff920a47034b08) 00000080:00200000:3.0:1788572250.116245:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x73:0x0](ffff920a470363c8) 00000080:00200000:3.0:1788572250.116255:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xd:0x0](ffff920a47036c08) 00000080:00200000:3.0:1788572250.116263:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xd:0x0](ffff920a47035348) 00000080:00200000:3.0:1788572250.116271:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x74:0x0](ffff920a47c5b248) 00000080:00200000:3.0:1788572250.116279:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x74:0x0](ffff920a47c5db88) 00000080:00200000:3.0:1788572250.116289:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xe:0x0](ffff920a48767448) 00000080:00200000:3.0:1788572250.116297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xe:0x0](ffff920a48763248) 00000080:00200000:3.0:1788572250.116306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x75:0x0](ffff920a470300c8) 00000080:00200000:3.0:1788572250.116316:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x75:0x0](ffff920a470321c8) 00000080:00200000:3.0:1788572250.116324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0xf:0x0](ffff920a48762a08) 00000080:00200000:3.0:1788572250.116332:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0xf:0x0](ffff920a487642c8) 00000080:00200000:3.0:1788572250.116340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x76:0x0](ffff920a47037448) 00000080:00200000:3.0:1788572250.116348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x76:0x0](ffff920a47035b88) 00000080:00200000:3.0:1788572250.116355:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x10:0x0](ffff920a47e800c8) 00000080:00200000:3.0:1788572250.116364:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x10:0x0](ffff920a48656c08) 00000080:00200000:3.0:1788572250.116373:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x77:0x0](ffff920a487663c8) 00000080:00200000:3.0:1788572250.116458:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x77:0x0](ffff920a471d8908) 00000080:00200000:3.0:1788572250.116470:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x11:0x0](ffff920a48765b88) 00000080:00200000:3.0:1788572250.116478:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x11:0x0](ffff920a48764b08) 00000080:00200000:3.0:1788572250.116486:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x78:0x0](ffff920a471de3c8) 00000080:00200000:3.0:1788572250.116495:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x78:0x0](ffff920a471df448) 00000080:00200000:3.0:1788572250.116502:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x12:0x0](ffff920a471dc2c8) 00000080:00200000:3.0:1788572250.116510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x12:0x0](ffff920a471daa08) 00000080:00200000:3.0:1788572250.116518:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x79:0x0](ffff920a48761988) 00000080:00200000:3.0:1788572250.116526:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x79:0x0](ffff920a48765348) 00000080:00200000:3.0:1788572250.116534:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x13:0x0](ffff920a48766c08) 00000080:00200000:3.0:1788572250.116541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x13:0x0](ffff920a487600c8) 00000080:00200000:3.0:1788572250.116549:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7a:0x0](ffff920a486500c8) 00000080:00200000:3.0:1788572250.116557:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7a:0x0](ffff920a47230908) 00000080:00200000:3.0:1788572250.116564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x14:0x0](ffff920a472363c8) 00000080:00200000:3.0:1788572250.116572:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x14:0x0](ffff920a472321c8) 00000080:00200000:3.0:1788572250.116581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7b:0x0](ffff920a47c5f448) 00000080:00200000:3.0:1788572250.116699:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7b:0x0](ffff920a47c59988) 00000080:00200000:3.0:1788572250.116709:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x15:0x0](ffff920a472342c8) 00000080:00200000:3.0:1788572250.116718:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x15:0x0](ffff920a47235b88) 00000080:00200000:3.0:1788572250.116728:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7c:0x0](ffff920a471ddb88) 00000080:00200000:3.0:1788572250.116738:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7c:0x0](ffff920a471dd348) 00000080:00200000:3.0:1788572250.116748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x16:0x0](ffff920a471d9148) 00000080:00200000:3.0:1788572250.116756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x16:0x0](ffff920a471dec08) 00000080:00200000:3.0:1788572250.116765:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7d:0x0](ffff920a47fe9988) 00000080:00200000:3.0:1788572250.116774:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7d:0x0](ffff920a47fe9148) 00000080:00200000:3.0:1788572250.116781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x17:0x0](ffff920a47236c08) 00000080:00200000:3.0:1788572250.116790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x17:0x0](ffff920a47235348) 00000080:00200000:3.0:1788572250.116798:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7e:0x0](ffff920a47239988) 00000080:00200000:3.0:1788572250.116807:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7e:0x0](ffff920a4723ba88) 00000080:00200000:3.0:1788572250.116815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x18:0x0](ffff920a47fee3c8) 00000080:00200000:3.0:1788572250.116831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x18:0x0](ffff920a47fedb88) 00000080:00200000:3.0:1788572250.116839:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x7f:0x0](ffff920a47c5a1c8) 00000080:00200000:3.0:1788572250.116847:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x7f:0x0](ffff920a47c580c8) 00000080:00200000:3.0:1788572250.116855:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x19:0x0](ffff920a471d9988) 00000080:00200000:3.0:1788572250.116863:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x19:0x0](ffff920a471dcb08) 00000080:00200000:3.0:1788572250.116870:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x80:0x0](ffff920a47fed348) 00000080:00200000:3.0:1788572250.116877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x80:0x0](ffff920a47fe80c8) 00000080:00200000:3.0:1788572250.116885:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1a:0x0](ffff920a471dba88) 00000080:00200000:3.0:1788572250.116894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1a:0x0](ffff920a471da1c8) 00000080:00200000:3.0:1788572250.116903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x81:0x0](ffff920a4723d348) 00000080:00200000:3.0:1788572250.116912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x81:0x0](ffff920a47238908) 00000080:00200000:3.0:1788572250.116921:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1b:0x0](ffff920a471d80c8) 00000080:00200000:3.0:1788572250.116940:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1b:0x0](ffff920a471db248) 00000080:00200000:3.0:1788572250.116949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x82:0x0](ffff920a4723c2c8) 00000080:00200000:3.0:1788572250.116957:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x82:0x0](ffff920a4723aa08) 00000080:00200000:3.0:1788572250.116966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1c:0x0](ffff920a4723f448) 00000080:00200000:3.0:1788572250.116975:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1c:0x0](ffff920a4723b248) 00000080:00200000:3.0:1788572250.116983:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x83:0x0](ffff920a48575348) 00000080:00200000:3.0:1788572250.116995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x83:0x0](ffff920a485742c8) 00000080:00200000:3.0:1788572250.117005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1d:0x0](ffff920a47feba88) 00000080:00200000:3.0:1788572250.117013:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1d:0x0](ffff920a471d1988) 00000080:00200000:3.0:1788572250.117021:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x84:0x0](ffff920a471d1148) 00000080:00200000:3.0:1788572250.117029:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x84:0x0](ffff920a471d21c8) 00000080:00200000:3.0:1788572250.117036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1e:0x0](ffff920a471d63c8) 00000080:00200000:3.0:1788572250.117043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1e:0x0](ffff920a471d7448) 00000080:00200000:3.0:1788572250.117052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x85:0x0](ffff920a471d00c8) 00000080:00200000:3.0:1788572250.117059:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x85:0x0](ffff920a471d3248) 00000080:00200000:3.0:1788572250.117067:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x1f:0x0](ffff920a471d3a88) 00000080:00200000:3.0:1788572250.117076:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x1f:0x0](ffff920a471d2a08) 00000080:00200000:3.0:1788572250.117084:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x86:0x0](ffff920a471e0908) 00000080:00200000:3.0:1788572250.117093:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x86:0x0](ffff920a471e21c8) 00000080:00200000:3.0:1788572250.117102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x20:0x0](ffff920a48573a88) 00000080:00200000:3.0:1788572250.117110:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x20:0x0](ffff920a48572a08) 00000080:00200000:3.0:1788572250.117197:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x87:0x0](ffff920a471e1148) 00000080:00200000:3.0:1788572250.117210:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x87:0x0](ffff920a471e3a88) 00000080:00200000:3.0:1788572250.117219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x21:0x0](ffff920a48574b08) 00000080:00200000:3.0:1788572250.117228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x21:0x0](ffff920a48570908) 00000080:00200000:3.0:1788572250.117243:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x88:0x0](ffff920a4716c2c8) 00000080:00200000:3.0:1788572250.117252:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x88:0x0](ffff920a4716ec08) 00000080:00200000:3.0:1788572250.117261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x22:0x0](ffff920a471680c8) 00000080:00200000:3.0:1788572250.117269:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x22:0x0](ffff920a4716db88) 00000080:00200000:3.0:1788572250.117279:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x89:0x0](ffff920a47043a88) 00000080:00200000:3.0:1788572250.117288:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x89:0x0](ffff920a470400c8) 00000080:00200000:3.0:1788572250.117297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x23:0x0](ffff920a471d4b08) 00000080:00200000:3.0:1788572250.117306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x23:0x0](ffff920a471d5348) 00000080:00200000:3.0:1788572250.117314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8a:0x0](ffff920a4716cb08) 00000080:00200000:3.0:1788572250.117325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8a:0x0](ffff920a4716aa08) 00000080:00200000:3.0:1788572250.117333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x24:0x0](ffff920a4716a1c8) 00000080:00200000:3.0:1788572250.117343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x24:0x0](ffff920a47169988) 00000080:00200000:3.0:1788572250.117352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8b:0x0](ffff920a4716ba88) 00000080:00200000:3.0:1788572250.117363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8b:0x0](ffff920a47169148) 00000080:00200000:3.0:1788572250.117372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x25:0x0](ffff920a4720aa08) 00000080:00200000:3.0:1788572250.117511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x25:0x0](ffff920a4720ec08) 00000080:00200000:3.0:1788572250.117525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8c:0x0](ffff920a47168908) 00000080:00200000:3.0:1788572250.117534:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8c:0x0](ffff920a4716e3c8) 00000080:00200000:3.0:1788572250.117542:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x26:0x0](ffff920a4720f448) 00000080:00200000:3.0:1788572250.117550:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x26:0x0](ffff920a47209988) 00000080:00200000:3.0:1788572250.117558:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8d:0x0](ffff920a47232a08) 00000080:00200000:3.0:1788572250.117568:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8d:0x0](ffff920a4716d348) 00000080:00200000:3.0:1788572250.117576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x27:0x0](ffff920a4720ba88) 00000080:00200000:3.0:1788572250.117695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x27:0x0](ffff920a4720cb08) 00000080:00200000:3.0:1788572250.117709:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8e:0x0](ffff920a4720c2c8) 00000080:00200000:3.0:1788572250.117716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8e:0x0](ffff920a4720e3c8) 00000080:00200000:3.0:1788572250.117725:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x28:0x0](ffff920a48576c08) 00000080:00200000:3.0:1788572250.117735:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x28:0x0](ffff920a48575b88) 00000080:00200000:3.0:1788572250.117743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x8f:0x0](ffff920a47042a08) 00000080:00200000:3.0:1788572250.117752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x8f:0x0](ffff920a47208908) 00000080:00200000:3.0:1788572250.117760:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x29:0x0](ffff920a48522a08) 00000080:00200000:3.0:1788572250.117769:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x29:0x0](ffff920a48525348) 00000080:00200000:3.0:1788572250.117778:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x90:0x0](ffff920a47ff21c8) 00000080:00200000:3.0:1788572250.117786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x90:0x0](ffff920a47ff42c8) 00000080:00200000:3.0:1788572250.117794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2a:0x0](ffff920a47105b88) 00000080:00200000:3.0:1788572250.117802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2a:0x0](ffff920a47106c08) 00000080:00200000:3.0:1788572250.117810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x91:0x0](ffff920a48523a88) 00000080:00200000:3.0:1788572250.117819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x91:0x0](ffff920a48525b88) 00000080:00200000:3.0:1788572250.117827:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2b:0x0](ffff920a47ff3248) 00000080:00200000:3.0:1788572250.117835:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2b:0x0](ffff920a47ff0908) 00000080:00200000:3.0:1788572250.117843:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x92:0x0](ffff920a47ff1148) 00000080:00200000:3.0:1788572250.117851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x92:0x0](ffff920a47ff2a08) 00000080:00200000:3.0:1788572250.117859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2c:0x0](ffff920a471e7448) 00000080:00200000:3.0:1788572250.117868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2c:0x0](ffff920a471e6c08) 00000080:00200000:3.0:1788572250.117876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x93:0x0](ffff920a471000c8) 00000080:00200000:3.0:1788572250.117884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x93:0x0](ffff920a47100908) 00000080:00200000:3.0:1788572250.117892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2d:0x0](ffff920a471021c8) 00000080:00200000:3.0:1788572250.117900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2d:0x0](ffff920a47107448) 00000080:00200000:3.0:1788572250.117908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x94:0x0](ffff920a47ff1988) 00000080:00200000:3.0:1788572250.117916:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x94:0x0](ffff920a47ff6c08) 00000080:00200000:3.0:1788572250.117925:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2e:0x0](ffff920a47ff00c8) 00000080:00200000:3.0:1788572250.117944:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2e:0x0](ffff920a47ff5b88) 00000080:00200000:3.0:1788572250.117953:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x95:0x0](ffff920a47ff3a88) 00000080:00200000:3.0:1788572250.117961:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x95:0x0](ffff920a47ff7448) 00000080:00200000:3.0:1788572250.117969:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x2f:0x0](ffff920a48526c08) 00000080:00200000:3.0:1788572250.117979:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x2f:0x0](ffff920a485242c8) 00000080:00200000:3.0:1788572250.117990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x96:0x0](ffff920a47eaa1c8) 00000080:00200000:3.0:1788572250.117999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x96:0x0](ffff920a47ea9988) 00000080:00200000:3.0:1788572250.118008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x30:0x0](ffff920a485221c8) 00000080:00200000:3.0:1788572250.118017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x30:0x0](ffff920a48527448) 00000080:00200000:3.0:1788572250.118025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x97:0x0](ffff920a48523248) 00000080:00200000:3.0:1788572250.118034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x97:0x0](ffff920a47ef21c8) 00000080:00200000:3.0:1788572250.118044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x31:0x0](ffff920a47ea9148) 00000080:00200000:3.0:1788572250.118053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x31:0x0](ffff920a47eaba88) 00000080:00200000:3.0:1788572250.118182:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x98:0x0](ffff920a47101988) 00000080:00200000:3.0:1788572250.118198:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x98:0x0](ffff920a47105348) 00000080:00200000:3.0:1788572250.118208:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x32:0x0](ffff920a47eadb88) 00000080:00200000:3.0:1788572250.118218:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x32:0x0](ffff920a47eaaa08) 00000080:00200000:3.0:1788572250.118226:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x99:0x0](ffff920a47e31148) 00000080:00200000:3.0:1788572250.118235:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x99:0x0](ffff920a47e342c8) 00000080:00200000:3.0:1788572250.118243:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x33:0x0](ffff920a47ef6c08) 00000080:00200000:3.0:1788572250.118251:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x33:0x0](ffff920a47ef1988) 00000080:00200000:3.0:1788572250.118259:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9a:0x0](ffff920a47ef4b08) 00000080:00200000:3.0:1788572250.118268:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9a:0x0](ffff920a47ef00c8) 00000080:00200000:3.0:1788572250.118276:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x34:0x0](ffff920a47eacb08) 00000080:00200000:3.0:1788572250.118284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x34:0x0](ffff920a47ea8908) 00000080:00200000:3.0:1788572250.118292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9b:0x0](ffff920a47ea80c8) 00000080:00200000:3.0:1788572250.118300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9b:0x0](ffff920a47eaf448) 00000080:00200000:3.0:1788572250.118308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x35:0x0](ffff920a47e34b08) 00000080:00200000:3.0:1788572250.118318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x35:0x0](ffff920a47e31988) 00000080:00200000:3.0:1788572250.118325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9c:0x0](ffff920a47ef5b88) 00000080:00200000:3.0:1788572250.118333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9c:0x0](ffff920a47ef7448) 00000080:00200000:3.0:1788572250.118343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x36:0x0](ffff920a47eac2c8) 00000080:00200000:3.0:1788572250.118351:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x36:0x0](ffff920a47eae3c8) 00000080:00200000:3.0:1788572250.118359:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9d:0x0](ffff920a47ef63c8) 00000080:00200000:3.0:1788572250.118367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9d:0x0](ffff920a47ef0908) 00000080:00200000:3.0:1788572250.118374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x37:0x0](ffff920a471e42c8) 00000080:00200000:3.0:1788572250.118455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x37:0x0](ffff920a471e5b88) 00000080:00200000:3.0:1788572250.118467:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9e:0x0](ffff920a47ff5348) 00000080:00200000:3.0:1788572250.118475:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9e:0x0](ffff920a47f4b248) 00000080:00200000:3.0:1788572250.118483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x38:0x0](ffff920a47f45b88) 00000080:00200000:3.0:1788572250.118492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x38:0x0](ffff920a47f442c8) 00000080:00200000:3.0:1788572250.118501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0x9f:0x0](ffff920a47f46c08) 00000080:00200000:3.0:1788572250.118509:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0x9f:0x0](ffff920a47f40908) 00000080:00200000:3.0:1788572250.118517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x39:0x0](ffff920a47f480c8) 00000080:00200000:3.0:1788572250.118525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x39:0x0](ffff920a47f4f448) 00000080:00200000:3.0:1788572250.118533:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa0:0x0](ffff920a47f463c8) 00000080:00200000:3.0:1788572250.118541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa0:0x0](ffff920a47f400c8) 00000080:00200000:3.0:1788572250.118548:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3a:0x0](ffff920a47f4ba88) 00000080:00200000:3.0:1788572250.118557:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3a:0x0](ffff920a47f4ec08) 00000080:00200000:3.0:1788572250.118565:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa1:0x0](ffff920a47f47448) 00000080:00200000:3.0:1788572250.118573:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa1:0x0](ffff920a47f42a08) 00000080:00200000:3.0:1788572250.118581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3b:0x0](ffff920a47f4d348) 00000080:00200000:3.0:1788572250.118704:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3b:0x0](ffff920a47f49988) 00000080:00200000:3.0:1788572250.118714:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa2:0x0](ffff920a47f421c8) 00000080:00200000:3.0:1788572250.118723:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa2:0x0](ffff920a47c45b88) 00000080:00200000:3.0:1788572250.118731:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3c:0x0](ffff920a47e37448) 00000080:00200000:3.0:1788572250.118739:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3c:0x0](ffff920a47e300c8) 00000080:00200000:3.0:1788572250.118747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa3:0x0](ffff920a47e363c8) 00000080:00200000:3.0:1788572250.118755:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa3:0x0](ffff920a47e36c08) 00000080:00200000:3.0:1788572250.118763:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3d:0x0](ffff920a47f48908) 00000080:00200000:3.0:1788572250.118771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3d:0x0](ffff920a47f4db88) 00000080:00200000:3.0:1788572250.118779:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa4:0x0](ffff920a48521988) 00000080:00200000:3.0:1788572250.118788:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa4:0x0](ffff920a487a5b88) 00000080:00200000:3.0:1788572250.118796:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3e:0x0](ffff920a47c47448) 00000080:00200000:3.0:1788572250.118804:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3e:0x0](ffff920a47c45348) 00000080:00200000:3.0:1788572250.118814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa5:0x0](ffff920a471e63c8) 00000080:00200000:3.0:1788572250.118822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa5:0x0](ffff920a471e4b08) 00000080:00200000:3.0:1788572250.118830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x3f:0x0](ffff920a47c46c08) 00000080:00200000:3.0:1788572250.118838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x3f:0x0](ffff920a47c43248) 00000080:00200000:3.0:1788572250.118846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa6:0x0](ffff920a487a2a08) 00000080:00200000:3.0:1788572250.118856:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa6:0x0](ffff920a487a42c8) 00000080:00200000:3.0:1788572250.118864:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x40:0x0](ffff920a47c41988) 00000080:00200000:3.0:1788572250.118872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x40:0x0](ffff920a47c43a88) 00000080:00200000:3.0:1788572250.118880:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa7:0x0](ffff920a471e5348) 00000080:00200000:3.0:1788572250.118888:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa7:0x0](ffff920a471e1988) 00000080:00200000:3.0:1788572250.118895:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x41:0x0](ffff920a487a3a88) 00000080:00200000:3.0:1788572250.118905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x41:0x0](ffff920a487a21c8) 00000080:00200000:3.0:1788572250.118913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa8:0x0](ffff920a487a1148) 00000080:00200000:3.0:1788572250.118921:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa8:0x0](ffff920a487a1988) 00000080:00200000:3.0:1788572250.118937:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x42:0x0](ffff920a471e00c8) 00000080:00200000:3.0:1788572250.118947:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x42:0x0](ffff920a471e2a08) 00000080:00200000:3.0:1788572250.118955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xa9:0x0](ffff920a47c44b08) 00000080:00200000:3.0:1788572250.118966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xa9:0x0](ffff920a47c463c8) 00000080:00200000:3.0:1788572250.118973:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x43:0x0](ffff920a47e35b88) 00000080:00200000:3.0:1788572250.118981:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x43:0x0](ffff920a47e321c8) 00000080:00200000:3.0:1788572250.118988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xaa:0x0](ffff920a471e3248) 00000080:00200000:3.0:1788572250.118997:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xaa:0x0](ffff920a47103248) 00000080:00200000:3.0:1788572250.119005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x44:0x0](ffff920a487a3248) 00000080:00200000:3.0:1788572250.119013:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x44:0x0](ffff920a487a63c8) 00000080:00200000:3.0:1788572250.119021:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xab:0x0](ffff920a487a0908) 00000080:00200000:3.0:1788572250.119030:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xab:0x0](ffff920a48659988) 00000080:00200000:3.0:1788572250.119039:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x45:0x0](ffff920a47ec00c8) 00000080:00200000:3.0:1788572250.119047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x45:0x0](ffff920a47ec1148) 00000080:00200000:3.0:1788572250.119054:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xac:0x0](ffff920a47ec2a08) 00000080:00200000:3.0:1788572250.119192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xac:0x0](ffff920a47ec21c8) 00000080:00200000:3.0:1788572250.119208:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x46:0x0](ffff920a47c5d348) 00000080:00200000:3.0:1788572250.119217:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x46:0x0](ffff920a4701aa08) 00000080:00200000:3.0:1788572250.119224:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xad:0x0](ffff920a47ec5b88) 00000080:00200000:3.0:1788572250.119232:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xad:0x0](ffff920a47ec63c8) 00000080:00200000:3.0:1788572250.119240:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x47:0x0](ffff920a47ec3a88) 00000080:00200000:3.0:1788572250.119249:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x47:0x0](ffff920a47ec1988) 00000080:00200000:3.0:1788572250.119256:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xae:0x0](ffff920a4865c2c8) 00000080:00200000:3.0:1788572250.119266:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xae:0x0](ffff920a4865aa08) 00000080:00200000:3.0:1788572250.119274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x48:0x0](ffff920a47ec7448) 00000080:00200000:3.0:1788572250.119282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x48:0x0](ffff920a47ec4b08) 00000080:00200000:3.0:1788572250.119289:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xaf:0x0](ffff920a47019988) 00000080:00200000:3.0:1788572250.119300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xaf:0x0](ffff920a4701b248) 00000080:00200000:3.0:1788572250.119308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x49:0x0](ffff920a486580c8) 00000080:00200000:3.0:1788572250.119318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x49:0x0](ffff920a4865ba88) 00000080:00200000:3.0:1788572250.119326:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb0:0x0](ffff920a4857aa08) 00000080:00200000:3.0:1788572250.119336:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb0:0x0](ffff920a48579148) 00000080:00200000:3.0:1788572250.119345:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4a:0x0](ffff920a4857ba88) 00000080:00200000:3.0:1788572250.119353:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4a:0x0](ffff920a48579988) 00000080:00200000:3.0:1788572250.119360:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb1:0x0](ffff920a47c400c8) 00000080:00200000:3.0:1788572250.119369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb1:0x0](ffff920a4865f448) 00000080:00200000:3.0:1788572250.119451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4b:0x0](ffff920a47e32a08) 00000080:00200000:3.0:1788572250.119465:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4b:0x0](ffff920a4703db88) 00000080:00200000:3.0:1788572250.119474:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb2:0x0](ffff920a47018908) 00000080:00200000:3.0:1788572250.119483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb2:0x0](ffff920a4701e3c8) 00000080:00200000:3.0:1788572250.119492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4c:0x0](ffff920a487ad348) 00000080:00200000:3.0:1788572250.119503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4c:0x0](ffff920a487aaa08) 00000080:00200000:3.0:1788572250.119514:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb3:0x0](ffff920a4857a1c8) 00000080:00200000:3.0:1788572250.119522:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb3:0x0](ffff920a4865d348) 00000080:00200000:3.0:1788572250.119531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4d:0x0](ffff920a4703aa08) 00000080:00200000:3.0:1788572250.119539:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4d:0x0](ffff920a4703e3c8) 00000080:00200000:3.0:1788572250.119547:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb4:0x0](ffff920a4701ba88) 00000080:00200000:3.0:1788572250.119555:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb4:0x0](ffff920a4701d348) 00000080:00200000:3.0:1788572250.119564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4e:0x0](ffff920a47c42a08) 00000080:00200000:3.0:1788572250.119574:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4e:0x0](ffff920a4705c2c8) 00000080:00200000:3.0:1788572250.119583:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb5:0x0](ffff920a4701cb08) 00000080:00200000:3.0:1788572250.119784:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb5:0x0](ffff920a4701c2c8) 00000080:00200000:3.0:1788572250.119793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x4f:0x0](ffff920a487a9988) 00000080:00200000:3.0:1788572250.119805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x4f:0x0](ffff920a487aa1c8) 00000080:00200000:3.0:1788572250.119814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb6:0x0](ffff920a47059148) 00000080:00200000:3.0:1788572250.119823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb6:0x0](ffff920a4705a1c8) 00000080:00200000:3.0:1788572250.119833:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x50:0x0](ffff920a487a9148) 00000080:00200000:3.0:1788572250.119842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x50:0x0](ffff920a487a80c8) 00000080:00200000:3.0:1788572250.119851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb7:0x0](ffff920a47059988) 00000080:00200000:3.0:1788572250.119859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb7:0x0](ffff920a47058908) 00000080:00200000:3.0:1788572250.119868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x51:0x0](ffff920a487aba88) 00000080:00200000:3.0:1788572250.119876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x51:0x0](ffff920a4857cb08) 00000080:00200000:3.0:1788572250.119884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb8:0x0](ffff920a4717a1c8) 00000080:00200000:3.0:1788572250.119892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb8:0x0](ffff920a47179148) 00000080:00200000:3.0:1788572250.119901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x52:0x0](ffff920a4717db88) 00000080:00200000:3.0:1788572250.119909:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x52:0x0](ffff920a4717b248) 00000080:00200000:3.0:1788572250.119916:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xb9:0x0](ffff920a47179988) 00000080:00200000:3.0:1788572250.119926:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xb9:0x0](ffff920a4717cb08) 00000080:00200000:3.0:1788572250.119945:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x53:0x0](ffff920a4705db88) 00000080:00200000:3.0:1788572250.119954:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x53:0x0](ffff920a4705ec08) 00000080:00200000:3.0:1788572250.119963:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xba:0x0](ffff920a47063248) 00000080:00200000:3.0:1788572250.119971:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xba:0x0](ffff920a470663c8) 00000080:00200000:3.0:1788572250.119979:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x54:0x0](ffff920a4717e3c8) 00000080:00200000:3.0:1788572250.119987:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x54:0x0](ffff920a4717ba88) 00000080:00200000:3.0:1788572250.119994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xbb:0x0](ffff920a4721d348) 00000080:00200000:3.0:1788572250.120002:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xbb:0x0](ffff920a4721a1c8) 00000080:00200000:3.0:1788572250.120011:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x55:0x0](ffff920a47061148) 00000080:00200000:3.0:1788572250.120022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x55:0x0](ffff920a47064b08) 00000080:00200000:3.0:1788572250.120032:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xbc:0x0](ffff920a47061988) 00000080:00200000:3.0:1788572250.120040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xbc:0x0](ffff920a47063a88) 00000080:00200000:3.0:1788572250.120049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x56:0x0](ffff920a4701db88) 00000080:00200000:3.0:1788572250.120058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x56:0x0](ffff920a4705ba88) 00000080:00200000:3.0:1788572250.120081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xbd:0x0](ffff920a47060908) 00000080:00200000:3.0:1788572250.120090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xbd:0x0](ffff920a470621c8) 00000080:00200000:3.0:1788572250.120099:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x57:0x0](ffff920a4703d348) 00000080:00200000:3.0:1788572250.120108:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x57:0x0](ffff920a4703ec08) 00000080:00200000:3.0:1788572250.120247:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xbe:0x0](ffff920a4703b248) 00000080:00200000:3.0:1788572250.120261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xbe:0x0](ffff920a4703c2c8) 00000080:00200000:3.0:1788572250.120268:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x58:0x0](ffff920a47065348) 00000080:00200000:3.0:1788572250.120277:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x58:0x0](ffff920a4701ec08) 00000080:00200000:3.0:1788572250.120284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xbf:0x0](ffff920a4721aa08) 00000080:00200000:3.0:1788572250.120292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xbf:0x0](ffff920a4721db88) 00000080:00200000:3.0:1788572250.120301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x59:0x0](ffff920a487680c8) 00000080:00200000:3.0:1788572250.120310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x59:0x0](ffff920a4876b248) 00000080:00200000:3.0:1788572250.120318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc0:0x0](ffff920a487ee3c8) 00000080:00200000:3.0:1788572250.120328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc0:0x0](ffff920a487eb248) 00000080:00200000:3.0:1788572250.120337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5a:0x0](ffff920a487ed348) 00000080:00200000:3.0:1788572250.120345:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5a:0x0](ffff920a487eec08) 00000080:00200000:3.0:1788572250.120353:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc1:0x0](ffff920a47219988) 00000080:00200000:3.0:1788572250.120362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc1:0x0](ffff920a4721c2c8) 00000080:00200000:3.0:1788572250.120370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5b:0x0](ffff920a4721ba88) 00000080:00200000:3.0:1788572250.120458:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5b:0x0](ffff920a4721ec08) 00000080:00200000:3.0:1788572250.120473:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc2:0x0](ffff920a4876ec08) 00000080:00200000:3.0:1788572250.120483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc2:0x0](ffff920a4876db88) 00000080:00200000:3.0:1788572250.120492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5c:0x0](ffff920a4876aa08) 00000080:00200000:3.0:1788572250.120501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5c:0x0](ffff920a4876cb08) 00000080:00200000:3.0:1788572250.120510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc3:0x0](ffff920a48769988) 00000080:00200000:3.0:1788572250.120519:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc3:0x0](ffff920a4876a1c8) 00000080:00200000:3.0:1788572250.120528:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5d:0x0](ffff920a472180c8) 00000080:00200000:3.0:1788572250.120537:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5d:0x0](ffff920a4721cb08) 00000080:00200000:3.0:1788572250.120568:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc4:0x0](ffff920a4721e3c8) 00000080:00200000:3.0:1788572250.120579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc4:0x0](ffff920a47178908) 00000080:00200000:3.0:1788572250.120716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5e:0x0](ffff920a4872f448) 00000080:00200000:3.0:1788572250.120729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5e:0x0](ffff920a487280c8) 00000080:00200000:3.0:1788572250.120739:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc5:0x0](ffff920a4876ba88) 00000080:00200000:3.0:1788572250.120748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc5:0x0](ffff920a48768908) 00000080:00200000:3.0:1788572250.120757:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x5f:0x0](ffff920a47038908) 00000080:00200000:3.0:1788572250.120766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x5f:0x0](ffff920a4703a1c8) 00000080:00200000:3.0:1788572250.120774:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc6:0x0](ffff920a4872ba88) 00000080:00200000:3.0:1788572250.120782:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc6:0x0](ffff920a4872c2c8) 00000080:00200000:3.0:1788572250.120791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x60:0x0](ffff920a4872ec08) 00000080:00200000:3.0:1788572250.120799:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x60:0x0](ffff920a48728908) 00000080:00200000:3.0:1788572250.120807:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc7:0x0](ffff920a4872db88) 00000080:00200000:3.0:1788572250.120816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc7:0x0](ffff920a4872e3c8) 00000080:00200000:3.0:1788572250.120824:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x61:0x0](ffff920a4872cb08) 00000080:00200000:3.0:1788572250.120832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x61:0x0](ffff920a4872b248) 00000080:00200000:3.0:1788572250.120840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc8:0x0](ffff920a4876e3c8) 00000080:00200000:3.0:1788572250.120851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc8:0x0](ffff920a4876d348) 00000080:00200000:3.0:1788572250.120859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x62:0x0](ffff920a4872d348) 00000080:00200000:3.0:1788572250.120868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x62:0x0](ffff920a48729988) 00000080:00200000:3.0:1788572250.120877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xc9:0x0](ffff920a487eba88) 00000080:00200000:3.0:1788572250.120887:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xc9:0x0](ffff920a487ec2c8) 00000080:00200000:3.0:1788572250.120896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x63:0x0](ffff920a487e8908) 00000080:00200000:3.0:1788572250.120906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x63:0x0](ffff920a487edb88) 00000080:00200000:3.0:1788572250.120915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xca:0x0](ffff920a4876f448) 00000080:00200000:3.0:1788572250.120923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xca:0x0](ffff920a4872aa08) 00000080:00200000:3.0:1788572250.120942:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x64:0x0](ffff920a485c7448) 00000080:00200000:3.0:1788572250.120952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x64:0x0](ffff920a485c2a08) 00000080:00200000:3.0:1788572250.120960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xcb:0x0](ffff920a485c4b08) 00000080:00200000:3.0:1788572250.120967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xcb:0x0](ffff920a485c0908) 00000080:00200000:3.0:1788572250.120976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x65:0x0](ffff920a485c1988) 00000080:00200000:3.0:1788572250.120985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x65:0x0](ffff920a485c42c8) 00000080:00200000:3.0:1788572250.120994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xcc:0x0](ffff920a4722a1c8) 00000080:00200000:3.0:1788572250.121002:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xcc:0x0](ffff920a4722b248) 00000080:00200000:3.0:1788572250.121011:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x66:0x0](ffff920a4722ec08) 00000080:00200000:3.0:1788572250.121022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x66:0x0](ffff920a4722d348) 00000080:00200000:3.0:1788572250.121031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xcd:0x0](ffff920a48634b08) 00000080:00200000:3.0:1788572250.121040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xcd:0x0](ffff920a486321c8) 00000080:00200000:3.0:1788572250.121050:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x67:0x0](ffff920a4722e3c8) 00000080:00200000:3.0:1788572250.121058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x67:0x0](ffff920a4722c2c8) 00000080:00200000:3.0:1788572250.121079:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xce:0x0](ffff920a48630908) 00000080:00200000:3.0:1788572250.121088:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xce:0x0](ffff920a48631988) 00000080:00200000:3.0:1788572250.121097:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x68:0x0](ffff920a48769148) 00000080:00200000:3.0:1788572250.121106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x68:0x0](ffff920a47228908) 00000080:00200000:3.0:1788572250.121115:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xcf:0x0](ffff920a486363c8) 00000080:00200000:3.0:1788572250.121243:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xcf:0x0](ffff920a486342c8) 00000080:00200000:3.0:1788572250.121252:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x69:0x0](ffff920a47fdba88) 00000080:00200000:3.0:1788572250.121261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x69:0x0](ffff920a47fd9148) 00000080:00200000:3.0:1788572250.121268:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd0:0x0](ffff920a4719a1c8) 00000080:00200000:3.0:1788572250.121278:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd0:0x0](ffff920a47199148) 00000080:00200000:3.0:1788572250.121287:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6a:0x0](ffff920a47fdec08) 00000080:00200000:3.0:1788572250.121318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6a:0x0](ffff920a47fdf448) 00000080:00200000:3.0:1788572250.121329:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd1:0x0](ffff920a4719b248) 00000080:00200000:3.0:1788572250.121339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd1:0x0](ffff920a471980c8) 00000080:00200000:3.0:1788572250.121347:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6b:0x0](ffff920a4703f448) 00000080:00200000:3.0:1788572250.121355:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6b:0x0](ffff920a470380c8) 00000080:00200000:3.0:1788572250.121363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd2:0x0](ffff920a47ec6c08) 00000080:00200000:3.0:1788572250.121371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd2:0x0](ffff920a47337448) 00000080:00200000:3.0:1788572250.121446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6c:0x0](ffff920a4719d348) 00000080:00200000:3.0:1788572250.121462:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6c:0x0](ffff920a4719cb08) 00000080:00200000:3.0:1788572250.121471:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd3:0x0](ffff920a487e9148) 00000080:00200000:3.0:1788572250.121482:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd3:0x0](ffff920a487ea1c8) 00000080:00200000:3.0:1788572250.121491:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6d:0x0](ffff920a47198908) 00000080:00200000:3.0:1788572250.121502:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6d:0x0](ffff920a4719c2c8) 00000080:00200000:3.0:1788572250.121511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd4:0x0](ffff920a47fd8908) 00000080:00200000:3.0:1788572250.121519:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd4:0x0](ffff920a47fdcb08) 00000080:00200000:3.0:1788572250.121527:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6e:0x0](ffff920a47fd00c8) 00000080:00200000:3.0:1788572250.121536:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6e:0x0](ffff920a47fd7448) 00000080:00200000:3.0:1788572250.121544:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd5:0x0](ffff920a47fd6c08) 00000080:00200000:3.0:1788572250.121551:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd5:0x0](ffff920a47fd4b08) 00000080:00200000:3.0:1788572250.121559:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x6f:0x0](ffff920a4719e3c8) 00000080:00200000:3.0:1788572250.121568:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x6f:0x0](ffff920a4719f448) 00000080:00200000:3.0:1788572250.121576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd6:0x0](ffff920a4719ec08) 00000080:00200000:3.0:1788572250.121583:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd6:0x0](ffff920a47fddb88) 00000080:00200000:3.0:1788572250.121711:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x70:0x0](ffff920a47fd63c8) 00000080:00200000:3.0:1788572250.121720:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x70:0x0](ffff920a47fd42c8) 00000080:00200000:3.0:1788572250.121728:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd7:0x0](ffff920a486300c8) 00000080:00200000:3.0:1788572250.121735:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd7:0x0](ffff920a4719ba88) 00000080:00200000:3.0:1788572250.121743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x71:0x0](ffff920a47273a88) 00000080:00200000:3.0:1788572250.121752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x71:0x0](ffff920a47276c08) 00000080:00200000:3.0:1788572250.121759:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd8:0x0](ffff920a4702db88) 00000080:00200000:3.0:1788572250.121768:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd8:0x0](ffff920a4702b248) 00000080:00200000:3.0:1788572250.121776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x72:0x0](ffff920a470280c8) 00000080:00200000:3.0:1788572250.121784:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x72:0x0](ffff920a4702a1c8) 00000080:00200000:3.0:1788572250.121791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xd9:0x0](ffff920a47028908) 00000080:00200000:3.0:1788572250.121797:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xd9:0x0](ffff920a4702ec08) 00000080:00200000:3.0:1788572250.121805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x73:0x0](ffff920a4702f448) 00000080:00200000:3.0:1788572250.121813:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x73:0x0](ffff920a4702d348) 00000080:00200000:3.0:1788572250.121822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xda:0x0](ffff920a487eaa08) 00000080:00200000:3.0:1788572250.121831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xda:0x0](ffff920a47218908) 00000080:00200000:3.0:1788572250.121841:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x74:0x0](ffff920a47ee8908) 00000080:00200000:3.0:1788572250.121850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x74:0x0](ffff920a47ee80c8) 00000080:00200000:3.0:1788572250.121858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xdb:0x0](ffff920a47ee4b08) 00000080:00200000:3.0:1788572250.121866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xdb:0x0](ffff920a47ee1988) 00000080:00200000:3.0:1788572250.121877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x75:0x0](ffff920a47eec2c8) 00000080:00200000:3.0:1788572250.121886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x75:0x0](ffff920a47eea1c8) 00000080:00200000:3.0:1788572250.121894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xdc:0x0](ffff920a47334b08) 00000080:00200000:3.0:1788572250.121904:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xdc:0x0](ffff920a47335b88) 00000080:00200000:3.0:1788572250.121912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x76:0x0](ffff920a47eee3c8) 00000080:00200000:3.0:1788572250.121920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x76:0x0](ffff920a47ee9148) 00000080:00200000:3.0:1788572250.121940:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xdd:0x0](ffff920a47ee42c8) 00000080:00200000:3.0:1788572250.121951:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xdd:0x0](ffff920a47ee00c8) 00000080:00200000:3.0:1788572250.121978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x77:0x0](ffff920a47eed348) 00000080:00200000:3.0:1788572250.121990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x77:0x0](ffff920a47ee9988) 00000080:00200000:3.0:1788572250.121997:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xde:0x0](ffff920a472742c8) 00000080:00200000:3.0:1788572250.122007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xde:0x0](ffff920a47271988) 00000080:00200000:3.0:1788572250.122016:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x78:0x0](ffff920a47ee21c8) 00000080:00200000:3.0:1788572250.122029:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x78:0x0](ffff920a47ee1148) 00000080:00200000:3.0:1788572250.122037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xdf:0x0](ffff920a472721c8) 00000080:00200000:3.0:1788572250.122046:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xdf:0x0](ffff920a47273248) 00000080:00200000:3.0:1788572250.122055:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x79:0x0](ffff920a47274b08) 00000080:00200000:3.0:1788572250.122192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x79:0x0](ffff920a47272a08) 00000080:00200000:3.0:1788572250.122205:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe0:0x0](ffff920a47eeba88) 00000080:00200000:3.0:1788572250.122214:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe0:0x0](ffff920a47eeaa08) 00000080:00200000:3.0:1788572250.122222:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7a:0x0](ffff920a47331148) 00000080:00200000:3.0:1788572250.122231:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7a:0x0](ffff920a47331988) 00000080:00200000:3.0:1788572250.122239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe1:0x0](ffff920a47270908) 00000080:00200000:3.0:1788572250.122248:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe1:0x0](ffff920a47275348) 00000080:00200000:3.0:1788572250.122257:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7b:0x0](ffff920a47ee63c8) 00000080:00200000:3.0:1788572250.122266:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7b:0x0](ffff920a47ee5b88) 00000080:00200000:3.0:1788572250.122274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe2:0x0](ffff920a47110908) 00000080:00200000:3.0:1788572250.122282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe2:0x0](ffff920a47116c08) 00000080:00200000:3.0:1788572250.122290:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7c:0x0](ffff920a47271148) 00000080:00200000:3.0:1788572250.122299:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7c:0x0](ffff920a47277448) 00000080:00200000:3.0:1788572250.122306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe3:0x0](ffff920a47ee7448) 00000080:00200000:3.0:1788572250.122315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe3:0x0](ffff920a47ee6c08) 00000080:00200000:3.0:1788572250.122323:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7d:0x0](ffff920a47111148) 00000080:00200000:3.0:1788572250.122332:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7d:0x0](ffff920a471163c8) 00000080:00200000:3.0:1788572250.122339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe4:0x0](ffff920a47199988) 00000080:00200000:3.0:1788572250.122349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe4:0x0](ffff920a47fd1148) 00000080:00200000:3.0:1788572250.122357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7e:0x0](ffff920a4711cb08) 00000080:00200000:3.0:1788572250.122365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7e:0x0](ffff920a4711e3c8) 00000080:00200000:3.0:1788572250.122372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe5:0x0](ffff920a47118908) 00000080:00200000:3.0:1788572250.122470:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe5:0x0](ffff920a4711db88) 00000080:00200000:3.0:1788572250.122482:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x7f:0x0](ffff920a47ee2a08) 00000080:00200000:3.0:1788572250.122489:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x7f:0x0](ffff920a4710a1c8) 00000080:00200000:3.0:1788572250.122499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe6:0x0](ffff920a471100c8) 00000080:00200000:3.0:1788572250.122507:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe6:0x0](ffff920a47113a88) 00000080:00200000:3.0:1788572250.122515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x80:0x0](ffff920a47112a08) 00000080:00200000:3.0:1788572250.122523:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x80:0x0](ffff920a471142c8) 00000080:00200000:3.0:1788572250.122531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe7:0x0](ffff920a47330908) 00000080:00200000:3.0:1788572250.122539:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe7:0x0](ffff920a473342c8) 00000080:00200000:3.0:1788572250.122547:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x81:0x0](ffff920a471121c8) 00000080:00200000:3.0:1788572250.122557:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x81:0x0](ffff920a47115b88) 00000080:00200000:3.0:1788572250.122564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe8:0x0](ffff920a4702e3c8) 00000080:00200000:3.0:1788572250.122573:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe8:0x0](ffff920a4700d348) 00000080:00200000:3.0:1788572250.122581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x82:0x0](ffff920a471180c8) 00000080:00200000:3.0:1788572250.122705:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x82:0x0](ffff920a4711f448) 00000080:00200000:3.0:1788572250.122716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xe9:0x0](ffff920a487b42c8) 00000080:00200000:3.0:1788572250.122726:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xe9:0x0](ffff920a487b2a08) 00000080:00200000:3.0:1788572250.122735:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x83:0x0](ffff920a487b5348) 00000080:00200000:3.0:1788572250.122744:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x83:0x0](ffff920a487b1148) 00000080:00200000:3.0:1788572250.122754:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xea:0x0](ffff920a4711b248) 00000080:00200000:3.0:1788572250.122779:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xea:0x0](ffff920a470a1988) 00000080:00200000:3.0:1788572250.122791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x84:0x0](ffff920a470a4b08) 00000080:00200000:3.0:1788572250.122800:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x84:0x0](ffff920a470a2a08) 00000080:00200000:3.0:1788572250.122808:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xeb:0x0](ffff920a473300c8) 00000080:00200000:3.0:1788572250.122818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xeb:0x0](ffff920a47335348) 00000080:00200000:3.0:1788572250.122826:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x85:0x0](ffff920a4700e3c8) 00000080:00200000:3.0:1788572250.122834:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x85:0x0](ffff920a4700a1c8) 00000080:00200000:3.0:1788572250.122843:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xec:0x0](ffff920a473363c8) 00000080:00200000:3.0:1788572250.122852:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xec:0x0](ffff920a47333a88) 00000080:00200000:3.0:1788572250.122859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x86:0x0](ffff920a47009148) 00000080:00200000:3.0:1788572250.122866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x86:0x0](ffff920a4700aa08) 00000080:00200000:3.0:1788572250.122873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xed:0x0](ffff920a470a5348) 00000080:00200000:3.0:1788572250.122881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xed:0x0](ffff920a470a6c08) 00000080:00200000:3.0:1788572250.122889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x87:0x0](ffff920a470a00c8) 00000080:00200000:3.0:1788572250.122897:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x87:0x0](ffff920a470a63c8) 00000080:00200000:3.0:1788572250.122904:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xee:0x0](ffff920a470a0908) 00000080:00200000:3.0:1788572250.122912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xee:0x0](ffff920a470a3248) 00000080:00200000:3.0:1788572250.122920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x88:0x0](ffff920a473321c8) 00000080:00200000:3.0:1788572250.122939:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x88:0x0](ffff920a47333248) 00000080:00200000:3.0:1788572250.122948:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xef:0x0](ffff920a470a5b88) 00000080:00200000:3.0:1788572250.122956:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xef:0x0](ffff920a470a1148) 00000080:00200000:3.0:1788572250.122964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x89:0x0](ffff920a4700c2c8) 00000080:00200000:3.0:1788572250.122973:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x89:0x0](ffff920a470080c8) 00000080:00200000:3.0:1788572250.122980:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf0:0x0](ffff920a487b7448) 00000080:00200000:3.0:1788572250.122988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf0:0x0](ffff920a47275b88) 00000080:00200000:3.0:1788572250.122996:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8a:0x0](ffff920a472cc2c8) 00000080:00200000:3.0:1788572250.123003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8a:0x0](ffff920a472c8908) 00000080:00200000:3.0:1788572250.123011:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf1:0x0](ffff920a470a7448) 00000080:00200000:3.0:1788572250.123019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf1:0x0](ffff920a470a42c8) 00000080:00200000:3.0:1788572250.123026:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8b:0x0](ffff920a487b4b08) 00000080:00200000:3.0:1788572250.123034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8b:0x0](ffff920a4709d348) 00000080:00200000:3.0:1788572250.123042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf2:0x0](ffff920a471080c8) 00000080:00200000:3.0:1788572250.123051:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf2:0x0](ffff920a4710d348) 00000080:00200000:3.0:1788572250.123173:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8c:0x0](ffff920a4709cb08) 00000080:00200000:3.0:1788572250.123188:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8c:0x0](ffff920a4709b248) 00000080:00200000:3.0:1788572250.123196:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf3:0x0](ffff920a472ca1c8) 00000080:00200000:3.0:1788572250.123205:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf3:0x0](ffff920a472cba88) 00000080:00200000:3.0:1788572250.123213:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8d:0x0](ffff920a47109988) 00000080:00200000:3.0:1788572250.123220:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8d:0x0](ffff920a4710db88) 00000080:00200000:3.0:1788572250.123229:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf4:0x0](ffff920a472ccb08) 00000080:00200000:3.0:1788572250.123237:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf4:0x0](ffff920a472c9988) 00000080:00200000:3.0:1788572250.123244:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8e:0x0](ffff920a4709e3c8) 00000080:00200000:3.0:1788572250.123254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8e:0x0](ffff920a4709ec08) 00000080:00200000:3.0:1788572250.123262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf5:0x0](ffff920a47099148) 00000080:00200000:3.0:1788572250.123270:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf5:0x0](ffff920a4709a1c8) 00000080:00200000:3.0:1788572250.123278:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x8f:0x0](ffff920a4709f448) 00000080:00200000:3.0:1788572250.123286:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x8f:0x0](ffff920a47099988) 00000080:00200000:3.0:1788572250.123293:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf6:0x0](ffff920a4710c2c8) 00000080:00200000:3.0:1788572250.123301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf6:0x0](ffff920a4710aa08) 00000080:00200000:3.0:1788572250.123308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x90:0x0](ffff920a47108908) 00000080:00200000:3.0:1788572250.123315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x90:0x0](ffff920a4710ec08) 00000080:00200000:3.0:1788572250.123337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf7:0x0](ffff920a47332a08) 00000080:00200000:3.0:1788572250.123348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf7:0x0](ffff920a47336c08) 00000080:00200000:3.0:1788572250.123356:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x91:0x0](ffff920a487200c8) 00000080:00200000:3.0:1788572250.123364:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x91:0x0](ffff920a48721148) 00000080:00200000:3.0:1788572250.123373:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf8:0x0](ffff920a48723a88) 00000080:00200000:3.0:1788572250.123452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf8:0x0](ffff920a48720908) 00000080:00200000:3.0:1788572250.123463:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x92:0x0](ffff920a472000c8) 00000080:00200000:3.0:1788572250.123472:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x92:0x0](ffff920a47201988) 00000080:00200000:3.0:1788572250.123480:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xf9:0x0](ffff920a47efd348) 00000080:00200000:3.0:1788572250.123488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xf9:0x0](ffff920a47ef8908) 00000080:00200000:3.0:1788572250.123496:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001788:0x93:0x0](ffff920a472c80c8) 00000080:00200000:3.0:1788572250.123503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df2:0x93:0x0](ffff920a472cf448) 00000080:00200000:3.0:1788572250.123511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200024df0:0xfa:0x0](ffff920a47159988) 00000080:00200000:3.0:1788572250.123519:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240000fb9:0xfa:0x0](ffff920a4715c2c8) 00000080:00200000:1.0:1788572250.202807:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.202811:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.202816:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.202819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a85:0x0](ffff920a473a5b88) 00000080:00200000:1.0:1788572250.202838:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a86:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.202842:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.202847:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a86:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.202851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a86:0x0](ffff920a473be3c8) 00000080:00200000:1.0:1788572250.202951:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a87:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.202956:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.202964:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a87:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.202968:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a87:0x0](ffff920a473c6c08) 00000080:00200000:1.0:1788572250.202993:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a88:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.202998:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203006:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a88:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a88:0x0](ffff920a473aaa08) 00000080:00200000:1.0:1788572250.203033:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a89:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203037:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203045:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a89:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a89:0x0](ffff920a473c5348) 00000080:00200000:1.0:1788572250.203071:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203075:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203082:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203086:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8a:0x0](ffff920a473bcb08) 00000080:00200000:1.0:1788572250.203107:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203112:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203119:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203123:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8b:0x0](ffff920a473adb88) 00000080:00200000:1.0:1788572250.203144:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203149:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203156:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203159:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8c:0x0](ffff920a473aec08) 00000080:00200000:1.0:1788572250.203177:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203181:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203186:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203190:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8d:0x0](ffff920a473ad348) 00000080:00200000:1.0:1788572250.203208:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203212:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203217:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203221:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8e:0x0](ffff920a473b00c8) 00000080:00200000:1.0:1788572250.203240:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a8f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203243:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203249:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a8f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203252:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a8f:0x0](ffff920a473c3248) 00000080:00200000:1.0:1788572250.203272:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a90:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203276:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203282:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a90:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a90:0x0](ffff920a473b63c8) 00000080:00200000:1.0:1788572250.203303:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a91:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203307:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203313:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a91:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203317:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a91:0x0](ffff920a473b4b08) 00000080:00200000:1.0:1788572250.203337:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a92:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203341:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203347:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a92:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203351:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a92:0x0](ffff920a473c7448) 00000080:00200000:1.0:1788572250.203377:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a93:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203383:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203390:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a93:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203394:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a93:0x0](ffff920a473b1988) 00000080:00200000:1.0:1788572250.203419:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a94:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203423:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203430:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a94:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203434:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a94:0x0](ffff920a473c00c8) 00000080:00200000:1.0:1788572250.203456:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a95:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203461:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203468:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a95:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203472:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a95:0x0](ffff920a473c63c8) 00000080:00200000:1.0:1788572250.203495:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a96:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203500:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203506:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a96:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a96:0x0](ffff920a473b9988) 00000080:00200000:1.0:1788572250.203532:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a97:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203537:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.203544:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a97:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.203547:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a97:0x0](ffff920a473c4b08) 00000080:00200000:1.0:1788572250.203571:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1a98:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572250.203576:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572250.206036:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1a98:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572250.206057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1a98:0x0](ffff920a473ba1c8) 00020000:00200000:3.0:1788572250.216290:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216328:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac2:0x0](ffff920a473d3248) 00000080:00200000:3.0:1788572250.216413:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216420:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216429:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac3:0x0](ffff920a473e00c8) 00000080:00200000:3.0:1788572250.216479:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216485:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216493:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac4:0x0](ffff920a473e80c8) 00000080:00200000:3.0:1788572250.216532:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216537:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216544:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216547:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac5:0x0](ffff920a473d7448) 00000080:00200000:3.0:1788572250.216571:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216576:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216752:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216760:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac6:0x0](ffff920a473e63c8) 00000080:00200000:3.0:1788572250.216789:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216794:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216803:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216808:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac7:0x0](ffff920a473e4b08) 00000080:00200000:3.0:1788572250.216831:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216836:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216842:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216847:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac8:0x0](ffff920a473e1988) 00000080:00200000:3.0:1788572250.216868:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ac9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216873:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216879:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ac9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ac9:0x0](ffff920a473e21c8) 00000080:00200000:3.0:1788572250.216904:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aca:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216909:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216916:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aca:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aca:0x0](ffff920a473e1148) 00000080:00200000:3.0:1788572250.216953:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1acb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216958:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.216965:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1acb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.216969:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1acb:0x0](ffff920a473ee3c8) 00000080:00200000:3.0:1788572250.216991:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1acc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.216996:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217003:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1acc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1acc:0x0](ffff920a473dc2c8) 00000080:00200000:3.0:1788572250.217275:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1acd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.217281:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217290:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1acd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1acd:0x0](ffff920a473e3a88) 00000080:00200000:3.0:1788572250.217321:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ace:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.217327:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217334:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ace:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ace:0x0](ffff920a473d00c8) 00000080:00200000:3.0:1788572250.217361:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1acf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.217366:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217373:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1acf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1acf:0x0](ffff920a473e0908) 00000080:00200000:3.0:1788572250.217400:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.217404:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217411:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217416:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad0:0x0](ffff920a473d63c8) 00000080:00200000:3.0:1788572250.217439:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.217444:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.217450:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.217454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad1:0x0](ffff920a473e42c8) 00000080:00200000:3.0:1788572250.218301:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.218312:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.218326:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.218331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad2:0x0](ffff920a473e2a08) 00000080:00200000:3.0:1788572250.218360:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.218365:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.218373:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.218377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad3:0x0](ffff920a473d4b08) 00000080:00200000:3.0:1788572250.218405:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.218410:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.218418:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.218422:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad4:0x0](ffff920a473d1988) 00000080:00200000:3.0:1788572250.218445:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.218450:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.218458:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.219254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad5:0x0](ffff920a473ecb08) 00000080:00200000:3.0:1788572250.219308:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.219315:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.219329:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.219333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad6:0x0](ffff920a473e5b88) 00000080:00200000:3.0:1788572250.219359:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.219364:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.219370:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.219375:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad7:0x0](ffff920a473daa08) 00000080:00200000:3.0:1788572250.219400:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.219404:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.219410:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.219414:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad8:0x0](ffff920a473e6c08) 00000080:00200000:3.0:1788572250.219436:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ad9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.219441:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.219447:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ad9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.219452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ad9:0x0](ffff920a473e9988) 00000080:00200000:3.0:1788572250.220306:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ada:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220317:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220334:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ada:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ada:0x0](ffff920a473d21c8) 00000080:00200000:3.0:1788572250.220372:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1adb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220378:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220385:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1adb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220389:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1adb:0x0](ffff920a473ddb88) 00000080:00200000:3.0:1788572250.220415:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1adc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220420:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220427:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1adc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220431:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1adc:0x0](ffff920a473d1148) 00000080:00200000:3.0:1788572250.220454:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1add:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220840:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220854:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1add:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1add:0x0](ffff920a473ea1c8) 00000080:00200000:3.0:1788572250.220890:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ade:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220895:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220902:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ade:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220907:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ade:0x0](ffff920a473d3a88) 00000080:00200000:3.0:1788572250.220942:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1adf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220948:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.220956:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1adf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.220960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1adf:0x0](ffff920a473e5348) 00000080:00200000:3.0:1788572250.220988:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.220994:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.221002:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.221006:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae0:0x0](ffff920a473d0908) 00000080:00200000:3.0:1788572250.221027:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.221032:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.221039:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.221043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae1:0x0](ffff920a473d42c8) 00000080:00200000:3.0:1788572250.221074:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.221080:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.221087:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.221091:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae2:0x0](ffff920a473dec08) 00000080:00200000:3.0:1788572250.224009:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224032:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224065:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224075:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae3:0x0](ffff920a473f3a88) 00000080:00200000:3.0:1788572250.224147:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224152:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224160:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224165:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae4:0x0](ffff920a473f0908) 00000080:00200000:3.0:1788572250.224191:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224197:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224205:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224208:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae5:0x0](ffff920a473dd348) 00000080:00200000:3.0:1788572250.224235:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224239:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224246:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224250:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae6:0x0](ffff920a473f42c8) 00000080:00200000:3.0:1788572250.224272:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224277:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224284:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224288:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae7:0x0](ffff920a473e3248) 00000080:00200000:3.0:1788572250.224318:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224323:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224331:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae8:0x0](ffff920a473e9148) 00000080:00200000:3.0:1788572250.224361:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1ae9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224366:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224373:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1ae9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1ae9:0x0](ffff920a473db248) 00000080:00200000:3.0:1788572250.224397:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aea:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224403:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224409:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aea:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224412:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aea:0x0](ffff920a473e7448) 00000080:00200000:3.0:1788572250.224434:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aeb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224438:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224445:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aeb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224449:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aeb:0x0](ffff920a473f2a08) 00000080:00200000:3.0:1788572250.224483:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aec:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224488:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224494:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aec:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aec:0x0](ffff920a473f5b88) 00000080:00200000:3.0:1788572250.224520:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aed:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224524:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224530:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aed:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224535:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aed:0x0](ffff920a473f6c08) 00000080:00200000:3.0:1788572250.224553:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aee:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224558:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224565:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aee:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224569:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aee:0x0](ffff920a473eba88) 00000080:00200000:3.0:1788572250.224834:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aef:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224840:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224849:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aef:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aef:0x0](ffff920a473fa1c8) 00000080:00200000:3.0:1788572250.224877:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af0:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224883:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.224889:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af0:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.224893:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af0:0x0](ffff920a473f9148) 00000080:00200000:3.0:1788572250.224914:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.224920:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227169:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af1:0x0](ffff920a473f5348) 00000080:00200000:3.0:1788572250.227267:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227278:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227303:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af2:0x0](ffff920a473f3248) 00000080:00200000:3.0:1788572250.227334:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227340:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227349:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227353:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af3:0x0](ffff920a473fba88) 00000080:00200000:3.0:1788572250.227386:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227392:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227399:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af4:0x0](ffff920a473f8908) 00000080:00200000:3.0:1788572250.227428:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227434:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227442:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af5:0x0](ffff920a473fc2c8) 00000080:00200000:3.0:1788572250.227499:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227505:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227514:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227518:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af6:0x0](ffff920a473faa08) 00000080:00200000:3.0:1788572250.227545:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227551:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227557:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af7:0x0](ffff920a473fdb88) 00000080:00200000:3.0:1788572250.227901:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227911:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227922:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227938:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af8:0x0](ffff920a473e8908) 00000080:00200000:3.0:1788572250.227976:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1af9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.227982:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.227990:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1af9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.227994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1af9:0x0](ffff920a473df448) 00000080:00200000:3.0:1788572250.228023:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1afa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228029:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228037:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1afa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228041:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1afa:0x0](ffff920a473fec08) 00000080:00200000:3.0:1788572250.228219:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1afb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228225:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228236:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1afb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228240:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1afb:0x0](ffff920a473f7448) 00000080:00200000:3.0:1788572250.228269:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1afc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228274:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228281:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1afc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1afc:0x0](ffff920a473f00c8) 00000080:00200000:3.0:1788572250.228306:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1afd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228311:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228318:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1afd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1afd:0x0](ffff920a473fd348) 00000080:00200000:3.0:1788572250.228345:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1afe:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228350:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228357:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1afe:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1afe:0x0](ffff920a473fb248) 00000080:00200000:3.0:1788572250.228386:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1aff:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228391:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228399:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1aff:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228403:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1aff:0x0](ffff920a473ec2c8) 00000080:00200000:3.0:1788572250.228426:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b00:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.228430:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.228437:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b00:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.228441:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b00:0x0](ffff920a473ff448) 00000080:00200000:3.0:1788572250.234976:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b01:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.234993:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235022:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b01:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235030:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b01:0x0](ffff920a473eaa08) 00000080:00200000:3.0:1788572250.235094:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b02:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235100:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235109:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b02:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235113:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b02:0x0](ffff920a473f80c8) 00000080:00200000:3.0:1788572250.235219:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b03:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235225:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235234:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b03:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235238:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b03:0x0](ffff920a473f63c8) 00000080:00200000:3.0:1788572250.235268:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b04:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235274:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235280:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b04:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b04:0x0](ffff920a473d80c8) 00000080:00200000:3.0:1788572250.235312:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b05:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235318:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235324:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b05:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b05:0x0](ffff920a473fe3c8) 00000080:00200000:3.0:1788572250.235349:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b06:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235354:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235360:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b06:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b06:0x0](ffff920a473fcb08) 00000080:00200000:3.0:1788572250.235387:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b07:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235392:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235399:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b07:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235402:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b07:0x0](ffff920a473f9988) 00000080:00200000:3.0:1788572250.235426:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b08:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.235431:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.235438:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b08:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.235442:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b08:0x0](ffff920a473de3c8) 00000080:00200000:3.0:1788572250.236278:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b09:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236288:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236301:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b09:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b09:0x0](ffff920a50804b08) 00000080:00200000:3.0:1788572250.236339:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236344:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236351:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236355:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0a:0x0](ffff920a50801988) 00000080:00200000:3.0:1788572250.236376:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236381:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236387:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236390:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0b:0x0](ffff920a473f4b08) 00000080:00200000:3.0:1788572250.236413:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236418:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236424:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0c:0x0](ffff920a473edb88) 00000080:00200000:3.0:1788572250.236451:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236456:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236712:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236718:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0d:0x0](ffff920a473dcb08) 00000080:00200000:3.0:1788572250.236749:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236754:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236762:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0e:0x0](ffff920a508021c8) 00000080:00200000:3.0:1788572250.236788:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b0f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236793:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236799:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b0f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236804:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b0f:0x0](ffff920a473f1988) 00000080:00200000:3.0:1788572250.236824:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236829:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236836:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b10:0x0](ffff920a473eec08) 00000080:00200000:3.0:1788572250.236865:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236870:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236876:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b11:0x0](ffff920a50801148) 00000080:00200000:3.0:1788572250.236901:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236905:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236911:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b12:0x0](ffff920a50803a88) 00000080:00200000:3.0:1788572250.236948:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236954:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236961:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.236965:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b13:0x0](ffff920a50800908) 00000080:00200000:3.0:1788572250.236987:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.236992:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.236999:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.237003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b14:0x0](ffff920a473ed348) 00000080:00200000:3.0:1788572250.237023:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.237028:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.237035:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.237039:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b15:0x0](ffff920a473eb248) 00000080:00200000:3.0:1788572250.237060:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b16:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.237065:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.237071:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b16:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.237075:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b16:0x0](ffff920a473f21c8) 00000080:00200000:3.0:1788572250.237098:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.237103:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.237111:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.238241:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b17:0x0](ffff920a473f1148) 00000080:00200000:3.0:1788572250.238299:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b18:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.238308:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.238325:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b18:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.238331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b18:0x0](ffff920a5080f448) 00000080:00200000:3.0:1788572250.238355:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b19:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.238359:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.238366:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b19:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.238371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b19:0x0](ffff920a508080c8) 00000080:00200000:3.0:1788572250.238392:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.238398:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.238404:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.238409:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1a:0x0](ffff920a473d9988) 00000080:00200000:3.0:1788572250.238436:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.238440:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.238447:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.238451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1b:0x0](ffff920a5080e3c8) 00000080:00200000:3.0:1788572250.239237:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239247:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239262:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1c:0x0](ffff920a473ef448) 00000080:00200000:3.0:1788572250.239308:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239313:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239322:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1d:0x0](ffff920a508042c8) 00000080:00200000:3.0:1788572250.239352:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239357:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239364:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1e:0x0](ffff920a50802a08) 00000080:00200000:3.0:1788572250.239387:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239391:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239397:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239401:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b1f:0x0](ffff920a50805b88) 00000080:00200000:3.0:1788572250.239420:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b20:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239424:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239431:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b20:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239435:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b20:0x0](ffff920a5080cb08) 00000080:00200000:3.0:1788572250.239720:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b21:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239728:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239738:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b21:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239742:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b21:0x0](ffff920a473da1c8) 00000080:00200000:3.0:1788572250.239775:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b22:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239780:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239786:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b22:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b22:0x0](ffff920a50806c08) 00000080:00200000:3.0:1788572250.239812:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b23:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239817:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239824:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b23:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239828:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b23:0x0](ffff920a50809988) 00000080:00200000:3.0:1788572250.239850:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b24:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239855:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239862:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b24:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b24:0x0](ffff920a5080a1c8) 00000080:00200000:3.0:1788572250.239889:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b25:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.239894:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.239901:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b25:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.239905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b25:0x0](ffff920a50813a88) 00000080:00200000:3.0:1788572250.240572:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240581:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240727:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240734:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b26:0x0](ffff920a508180c8) 00000080:00200000:3.0:1788572250.240777:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240782:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240790:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b27:0x0](ffff920a5081e3c8) 00000080:00200000:3.0:1788572250.240817:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b28:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240823:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240830:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b28:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240833:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b28:0x0](ffff920a50809148) 00000080:00200000:3.0:1788572250.240858:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b29:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240863:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240870:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b29:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240874:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b29:0x0](ffff920a50805348) 00000080:00200000:3.0:1788572250.240899:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240904:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240911:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2a:0x0](ffff920a50803248) 00000080:00200000:3.0:1788572250.240951:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240956:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.240964:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.240968:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2b:0x0](ffff920a50807448) 00000080:00200000:3.0:1788572250.240989:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.240993:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241000:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.241003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2c:0x0](ffff920a508000c8) 00000080:00200000:3.0:1788572250.241025:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.241030:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241038:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.241043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2d:0x0](ffff920a50810908) 00000080:00200000:3.0:1788572250.241070:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.241076:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241084:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.241088:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2e:0x0](ffff920a508142c8) 00000080:00200000:3.0:1788572250.241113:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.241856:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241871:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.241877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b2f:0x0](ffff920a508063c8) 00000080:00200000:3.0:1788572250.241926:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.241941:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241951:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.241955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b30:0x0](ffff920a5080ba88) 00000080:00200000:3.0:1788572250.241984:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.241989:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.241996:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.242000:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b31:0x0](ffff920a50808908) 00000080:00200000:3.0:1788572250.242025:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.242030:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.242038:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.242042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b32:0x0](ffff920a50812a08) 00000080:00200000:3.0:1788572250.242073:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.242079:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.242086:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.242090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b33:0x0](ffff920a5080c2c8) 00000080:00200000:3.0:1788572250.242113:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243201:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243222:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b34:0x0](ffff920a5081cb08) 00000080:00200000:3.0:1788572250.243278:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243283:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243291:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243296:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b35:0x0](ffff920a5080aa08) 00000080:00200000:3.0:1788572250.243320:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243326:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243333:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b36:0x0](ffff920a5080db88) 00000080:00200000:3.0:1788572250.243360:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243365:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243371:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243375:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b37:0x0](ffff920a50819988) 00000080:00200000:3.0:1788572250.243398:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243404:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243412:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243416:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b38:0x0](ffff920a5081a1c8) 00000080:00200000:3.0:1788572250.243438:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243443:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243450:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b39:0x0](ffff920a5080ec08) 00000080:00200000:3.0:1788572250.243751:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243757:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243768:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243773:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3a:0x0](ffff920a5080d348) 00000080:00200000:3.0:1788572250.243796:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.243801:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.243808:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.243813:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3b:0x0](ffff920a50815b88) 00000080:00200000:3.0:1788572250.243995:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.244000:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.244008:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.244012:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3c:0x0](ffff920a5080b248) 00000080:00200000:3.0:1788572250.244037:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.244044:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.244051:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.244055:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3d:0x0](ffff920a50825348) 00000080:00200000:3.0:1788572250.244084:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.244089:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.244095:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.244099:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3e:0x0](ffff920a5082f448) 00000080:00200000:3.0:1788572250.245145:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245158:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245175:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245182:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b3f:0x0](ffff920a50816c08) 00000080:00200000:3.0:1788572250.245223:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b40:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245229:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245236:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b40:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245240:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b40:0x0](ffff920a508280c8) 00000080:00200000:3.0:1788572250.245264:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b41:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245269:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245276:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b41:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245280:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b41:0x0](ffff920a50823248) 00000080:00200000:3.0:1788572250.245307:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b42:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245313:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245320:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b42:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b42:0x0](ffff920a50819148) 00000080:00200000:3.0:1788572250.245355:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b43:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245359:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245366:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b43:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b43:0x0](ffff920a50827448) 00000080:00200000:3.0:1788572250.245395:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b44:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245401:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245407:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b44:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b44:0x0](ffff920a508200c8) 00000080:00200000:3.0:1788572250.245433:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b45:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245438:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245445:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b45:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245449:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b45:0x0](ffff920a5082e3c8) 00000080:00200000:3.0:1788572250.245494:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b46:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245500:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245507:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b46:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b46:0x0](ffff920a508263c8) 00000080:00200000:3.0:1788572250.245535:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b47:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245540:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.245547:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b47:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.245551:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b47:0x0](ffff920a5081ba88) 00000080:00200000:3.0:1788572250.245575:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b48:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.245580:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246557:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b48:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246566:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b48:0x0](ffff920a50824b08) 00000080:00200000:3.0:1788572250.246768:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b49:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246776:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246786:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b49:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b49:0x0](ffff920a50815348) 00000080:00200000:3.0:1788572250.246816:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246822:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246829:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246833:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4a:0x0](ffff920a50821988) 00000080:00200000:3.0:1788572250.246854:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246859:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246866:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246870:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4b:0x0](ffff920a5082cb08) 00000080:00200000:3.0:1788572250.246895:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246899:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246906:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246910:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4c:0x0](ffff920a508221c8) 00000080:00200000:3.0:1788572250.246947:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246953:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.246961:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.246965:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4d:0x0](ffff920a50818908) 00000080:00200000:3.0:1788572250.246994:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.246999:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247006:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4e:0x0](ffff920a50829988) 00000080:00200000:3.0:1788572250.247033:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b4f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.247037:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247044:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b4f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247048:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b4f:0x0](ffff920a5082a1c8) 00000080:00200000:3.0:1788572250.247070:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b50:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.247075:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247081:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b50:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247085:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b50:0x0](ffff920a50821148) 00000080:00200000:3.0:1788572250.247107:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b51:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.247112:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247869:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b51:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b51:0x0](ffff920a50829148) 00000080:00200000:3.0:1788572250.247914:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b52:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.247920:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247942:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b52:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247947:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b52:0x0](ffff920a50813248) 00000080:00200000:3.0:1788572250.247971:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b53:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.247977:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.247984:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b53:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.247988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b53:0x0](ffff920a5081c2c8) 00000080:00200000:3.0:1788572250.248015:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b54:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248020:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248027:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b54:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b54:0x0](ffff920a50817448) 00000080:00200000:3.0:1788572250.248052:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b55:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248056:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248062:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b55:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b55:0x0](ffff920a5081aa08) 00000080:00200000:3.0:1788572250.248089:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b56:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248094:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248101:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b56:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b56:0x0](ffff920a5082ba88) 00000080:00200000:3.0:1788572250.248582:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b57:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248682:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248694:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b57:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248699:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b57:0x0](ffff920a50828908) 00000080:00200000:3.0:1788572250.248727:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b58:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248732:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248740:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b58:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248744:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b58:0x0](ffff920a50823a88) 00000080:00200000:3.0:1788572250.248771:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b59:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248777:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248784:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b59:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b59:0x0](ffff920a50820908) 00000080:00200000:3.0:1788572250.248809:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248813:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248820:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248824:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5a:0x0](ffff920a5082c2c8) 00000080:00200000:3.0:1788572250.248847:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248852:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248860:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248864:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5b:0x0](ffff920a508100c8) 00000080:00200000:3.0:1788572250.248889:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248894:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248901:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5c:0x0](ffff920a5082aa08) 00000080:00200000:3.0:1788572250.248938:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248943:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248951:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5d:0x0](ffff920a508242c8) 00000080:00200000:3.0:1788572250.248978:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.248984:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.248991:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.248994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5e:0x0](ffff920a5082db88) 00000080:00200000:3.0:1788572250.249015:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b5f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.249020:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.249027:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b5f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.249031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b5f:0x0](ffff920a508163c8) 00000080:00200000:3.0:1788572250.249053:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b60:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.249058:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.249066:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b60:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.249070:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b60:0x0](ffff920a5081db88) 00000080:00200000:3.0:1788572250.249095:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b61:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.249100:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.249107:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b61:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.249111:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b61:0x0](ffff920a50822a08) 00000080:00200000:3.0:1788572250.251855:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b62:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.251869:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.251889:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b62:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.251897:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b62:0x0](ffff920a50814b08) 00000080:00200000:3.0:1788572250.252672:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b63:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252683:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252698:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b63:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252704:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b63:0x0](ffff920a50825b88) 00000080:00200000:3.0:1788572250.252744:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b64:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252749:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252757:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b64:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252762:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b64:0x0](ffff920a50826c08) 00000080:00200000:3.0:1788572250.252785:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b65:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252790:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252797:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b65:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b65:0x0](ffff920a5082ec08) 00000080:00200000:3.0:1788572250.252828:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b66:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252833:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252840:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b66:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252845:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b66:0x0](ffff920a50833248) 00000080:00200000:3.0:1788572250.252867:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b67:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252873:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252879:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b67:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252883:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b67:0x0](ffff920a5082d348) 00000080:00200000:3.0:1788572250.252919:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b68:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252924:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252944:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b68:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b68:0x0](ffff920a5082b248) 00000080:00200000:3.0:1788572250.252975:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b69:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.252980:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.252987:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b69:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.252991:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b69:0x0](ffff920a50837448) 00000080:00200000:3.0:1788572250.253015:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253020:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253026:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253030:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6a:0x0](ffff920a5081ec08) 00000080:00200000:3.0:1788572250.253057:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253062:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253069:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6b:0x0](ffff920a5081d348) 00000080:00200000:3.0:1788572250.253093:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253098:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253105:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253109:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6c:0x0](ffff920a5083ec08) 00000080:00200000:3.0:1788572250.253131:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253136:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253142:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253147:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6d:0x0](ffff920a5083d348) 00000080:00200000:3.0:1788572250.253168:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253173:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253181:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253185:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6e:0x0](ffff920a50811988) 00000080:00200000:3.0:1788572250.253209:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b6f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253214:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253221:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b6f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b6f:0x0](ffff920a508121c8) 00000080:00200000:3.0:1788572250.253247:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b70:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253251:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253258:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b70:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b70:0x0](ffff920a508300c8) 00000080:00200000:3.0:1788572250.253287:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b71:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253292:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253298:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b71:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253302:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b71:0x0](ffff920a508363c8) 00000080:00200000:3.0:1788572250.253324:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b72:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253329:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253335:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b72:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b72:0x0](ffff920a5083b248) 00000080:00200000:3.0:1788572250.253361:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b73:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253366:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253373:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b73:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b73:0x0](ffff920a50834b08) 00000080:00200000:3.0:1788572250.253399:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b74:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253403:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253410:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b74:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253413:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b74:0x0](ffff920a5083f448) 00000080:00200000:3.0:1788572250.253502:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b75:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253509:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253516:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b75:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253520:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b75:0x0](ffff920a50831988) 00000080:00200000:3.0:1788572250.253550:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b76:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253555:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253563:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b76:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b76:0x0](ffff920a50811148) 00000080:00200000:3.0:1788572250.253610:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b77:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253616:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253623:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b77:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253627:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b77:0x0](ffff920a5081b248) 00000080:00200000:3.0:1788572250.253651:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b78:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253656:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253663:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b78:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253667:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b78:0x0](ffff920a508380c8) 00000080:00200000:3.0:1788572250.253690:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b79:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253695:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253701:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b79:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253705:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b79:0x0](ffff920a5081f448) 00000080:00200000:3.0:1788572250.253726:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253730:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253737:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253741:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7a:0x0](ffff920a5083e3c8) 00000080:00200000:3.0:1788572250.253762:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253766:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253773:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7b:0x0](ffff920a5083cb08) 00000080:00200000:3.0:1788572250.253799:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253804:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253810:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7c:0x0](ffff920a50839988) 00000080:00200000:3.0:1788572250.253835:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253840:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253847:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7d:0x0](ffff920a5083a1c8) 00000080:00200000:3.0:1788572250.253872:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253877:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253883:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253888:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7e:0x0](ffff920a50839148) 00000080:00200000:3.0:1788572250.253968:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b7f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.253975:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.253983:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b7f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.253988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b7f:0x0](ffff920a50845348) 00000080:00200000:3.0:1788572250.254015:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b80:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254020:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254027:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b80:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b80:0x0](ffff920a5083ba88) 00000080:00200000:3.0:1788572250.254054:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b81:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254059:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254066:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b81:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254070:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b81:0x0](ffff920a50843248) 00000080:00200000:3.0:1788572250.254094:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b82:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254099:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254106:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b82:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254110:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b82:0x0](ffff920a508321c8) 00000080:00200000:3.0:1788572250.254139:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b83:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254145:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254152:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b83:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254157:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b83:0x0](ffff920a50838908) 00000080:00200000:3.0:1788572250.254209:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b84:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254214:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254221:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b84:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254226:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b84:0x0](ffff920a5083c2c8) 00000080:00200000:3.0:1788572250.254248:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b85:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254253:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254259:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b85:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254263:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b85:0x0](ffff920a50847448) 00000080:00200000:3.0:1788572250.254292:0:160601:0:(vvp_io.c:1826:vvp_io_init()) [0x20001a9e3:0x1b86:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572250.254297:0:160601:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572250.254305:0:160601:0:(vvp_io.c:304:vvp_io_fini()) [0x20001a9e3:0x1b86:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572250.254310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b86:0x0](ffff920a5083aa08) 00000080:00200000:3.0:1788572250.254343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b87:0x0](ffff920a50831148) 00000080:00200000:3.0:1788572250.254357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x20001a9e3:0x1b8b:0x0](ffff920a50830908) 00000002:00100000:3.0:1788572250.254412:0:160601:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/608 e 0 to 0 dl 1788572243 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572250.254425:0:160601:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a42f86800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572250.254432:0:160601:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ca80 x1875444569483904/t0(0) o35->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572250.254454:0:160601:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job umount:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:160601:1875444569483904:192.168.202.157@tcp:35:umount.0 00000100:00100000:3.0:1788572250.254650:0:160601:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4805e840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572250.267739:0:160601:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job umount:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:160601:1875444569483904:192.168.202.157@tcp:35:umount.0 00000002:00100000:1.0:1788572250.267779:0:160601:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac6680 x1875444567597824/t326417519807(326417519807) o101->lustre-MDT0000-mdc-ffff920a42f86800@192.168.202.157@tcp:12/10 lens 520/608 e 0 to 0 dl 1788572243 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572250.267830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11:0x0](ffff920a5084a1c8) 00000080:00200000:1.0:1788572250.267850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11:0x0](ffff920a50849148) 00000080:00200000:1.0:1788572250.267858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12:0x0](ffff920a5084ba88) 00000080:00200000:1.0:1788572250.267866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12:0x0](ffff920a50848908) 00000080:00200000:1.0:1788572250.267874:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13:0x0](ffff920a50852a08) 00000080:00200000:1.0:1788572250.267882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13:0x0](ffff920a50855b88) 00000080:00200000:1.0:1788572250.267889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14:0x0](ffff920a50835b88) 00000080:00200000:1.0:1788572250.267896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14:0x0](ffff920a50836c08) 00000080:00200000:1.0:1788572250.267906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15:0x0](ffff920a508400c8) 00000080:00200000:1.0:1788572250.267913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15:0x0](ffff920a508463c8) 00000080:00200000:1.0:1788572250.267922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16:0x0](ffff920a50835348) 00000080:00200000:1.0:1788572250.267943:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16:0x0](ffff920a50832a08) 00000080:00200000:1.0:1788572250.267952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17:0x0](ffff920a5084c2c8) 00000080:00200000:1.0:1788572250.267961:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17:0x0](ffff920a5084aa08) 00000080:00200000:1.0:1788572250.267970:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18:0x0](ffff920a50856c08) 00000080:00200000:1.0:1788572250.267978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18:0x0](ffff920a50855348) 00000080:00200000:1.0:1788572250.267986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19:0x0](ffff920a50853248) 00000080:00200000:1.0:1788572250.267995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19:0x0](ffff920a50857448) 00000080:00200000:1.0:1788572250.268003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a:0x0](ffff920a508563c8) 00000080:00200000:1.0:1788572250.268010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a:0x0](ffff920a50854b08) 00000080:00200000:1.0:1788572250.268019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b:0x0](ffff920a508342c8) 00000080:00200000:1.0:1788572250.268028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b:0x0](ffff920a50833a88) 00000080:00200000:1.0:1788572250.268036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c:0x0](ffff920a50851148) 00000080:00200000:1.0:1788572250.268044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c:0x0](ffff920a50853a88) 00000080:00200000:1.0:1788572250.268053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d:0x0](ffff920a50858908) 00000080:00200000:1.0:1788572250.268061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d:0x0](ffff920a5085c2c8) 00000080:00200000:1.0:1788572250.268069:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e:0x0](ffff920a5084db88) 00000080:00200000:1.0:1788572250.268077:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e:0x0](ffff920a5084ec08) 00000080:00200000:1.0:1788572250.268085:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f:0x0](ffff920a5084d348) 00000080:00200000:1.0:1788572250.268092:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f:0x0](ffff920a5084b248) 00000080:00200000:1.0:1788572250.268101:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20:0x0](ffff920a50850908) 00000080:00200000:1.0:1788572250.268108:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20:0x0](ffff920a508542c8) 00000080:00200000:1.0:1788572250.268128:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21:0x0](ffff920a5085ec08) 00000080:00200000:1.0:1788572250.268138:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21:0x0](ffff920a5085d348) 00000080:00200000:1.0:1788572250.268147:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22:0x0](ffff920a5085f448) 00000080:00200000:1.0:1788572250.268155:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22:0x0](ffff920a508580c8) 00000080:00200000:1.0:1788572250.268163:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23:0x0](ffff920a508521c8) 00000080:00200000:1.0:1788572250.268171:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23:0x0](ffff920a50851988) 00000080:00200000:1.0:1788572250.268179:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24:0x0](ffff920a5085e3c8) 00000080:00200000:1.0:1788572250.268187:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24:0x0](ffff920a5085cb08) 00000080:00200000:1.0:1788572250.268195:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25:0x0](ffff920a50862a08) 00000080:00200000:1.0:1788572250.268203:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25:0x0](ffff920a50865b88) 00000080:00200000:1.0:1788572250.268210:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26:0x0](ffff920a5085b248) 00000080:00200000:1.0:1788572250.268218:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26:0x0](ffff920a50859988) 00000080:00200000:1.0:1788572250.268226:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27:0x0](ffff920a50865348) 00000080:00200000:1.0:1788572250.268234:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27:0x0](ffff920a50863248) 00000080:00200000:1.0:1788572250.268242:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28:0x0](ffff920a50867448) 00000080:00200000:1.0:1788572250.268250:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28:0x0](ffff920a508600c8) 00000080:00200000:1.0:1788572250.268258:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29:0x0](ffff920a50859148) 00000080:00200000:1.0:1788572250.268267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29:0x0](ffff920a5085ba88) 00000080:00200000:1.0:1788572250.268276:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a:0x0](ffff920a5085db88) 00000080:00200000:1.0:1788572250.268284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a:0x0](ffff920a5085aa08) 00000080:00200000:1.0:1788572250.268291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b:0x0](ffff920a5085a1c8) 00000080:00200000:1.0:1788572250.268299:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b:0x0](ffff920a5083db88) 00000080:00200000:1.0:1788572250.268307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c:0x0](ffff920a50864b08) 00000080:00200000:1.0:1788572250.268314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c:0x0](ffff920a50861988) 00000080:00200000:1.0:1788572250.268322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d:0x0](ffff920a5086f448) 00000080:00200000:1.0:1788572250.268330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d:0x0](ffff920a508680c8) 00000080:00200000:1.0:1788572250.268337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e:0x0](ffff920a5086e3c8) 00000080:00200000:1.0:1788572250.268345:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e:0x0](ffff920a5086cb08) 00000080:00200000:1.0:1788572250.268352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f:0x0](ffff920a508480c8) 00000080:00200000:1.0:1788572250.268360:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f:0x0](ffff920a5084e3c8) 00000080:00200000:1.0:1788572250.268367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30:0x0](ffff920a5086a1c8) 00000080:00200000:1.0:1788572250.268375:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30:0x0](ffff920a50869148) 00000080:00200000:1.0:1788572250.268384:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31:0x0](ffff920a50861148) 00000080:00200000:1.0:1788572250.268392:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31:0x0](ffff920a50863a88) 00000080:00200000:1.0:1788572250.268399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32:0x0](ffff920a50841148) 00000080:00200000:1.0:1788572250.268406:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32:0x0](ffff920a50843a88) 00000080:00200000:1.0:1788572250.268414:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33:0x0](ffff920a50860908) 00000080:00200000:1.0:1788572250.268421:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33:0x0](ffff920a508642c8) 00000080:00200000:1.0:1788572250.268428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34:0x0](ffff920a508621c8) 00000080:00200000:1.0:1788572250.268436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34:0x0](ffff920a508663c8) 00000080:00200000:1.0:1788572250.268444:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35:0x0](ffff920a50866c08) 00000080:00200000:1.0:1788572250.268451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35:0x0](ffff920a508500c8) 00000080:00200000:1.0:1788572250.268458:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36:0x0](ffff920a5086aa08) 00000080:00200000:1.0:1788572250.268466:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36:0x0](ffff920a5086db88) 00000080:00200000:1.0:1788572250.268474:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37:0x0](ffff920a5086ec08) 00000080:00200000:1.0:1788572250.268482:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37:0x0](ffff920a5086d348) 00000080:00200000:1.0:1788572250.268488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38:0x0](ffff920a50874b08) 00000080:00200000:1.0:1788572250.268496:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38:0x0](ffff920a50871988) 00000080:00200000:1.0:1788572250.268503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39:0x0](ffff920a5086c2c8) 00000080:00200000:1.0:1788572250.268510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39:0x0](ffff920a50868908) 00000080:00200000:1.0:1788572250.268517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a:0x0](ffff920a50849988) 00000080:00200000:1.0:1788572250.268525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a:0x0](ffff920a5084cb08) 00000080:00200000:1.0:1788572250.268533:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b:0x0](ffff920a50873a88) 00000080:00200000:1.0:1788572250.268541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b:0x0](ffff920a50870908) 00000080:00200000:1.0:1788572250.268548:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c:0x0](ffff920a508742c8) 00000080:00200000:1.0:1788572250.268555:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c:0x0](ffff920a50872a08) 00000080:00200000:1.0:1788572250.268563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d:0x0](ffff920a50869988) 00000080:00200000:1.0:1788572250.268571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d:0x0](ffff920a5086b248) 00000080:00200000:1.0:1788572250.268577:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e:0x0](ffff920a5086ba88) 00000080:00200000:1.0:1788572250.268609:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e:0x0](ffff920a5087ba88) 00000080:00200000:1.0:1788572250.268619:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f:0x0](ffff920a50878908) 00000080:00200000:1.0:1788572250.268627:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f:0x0](ffff920a5087c2c8) 00000080:00200000:1.0:1788572250.268635:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40:0x0](ffff920a50876c08) 00000080:00200000:1.0:1788572250.268644:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40:0x0](ffff920a50875348) 00000080:00200000:1.0:1788572250.268651:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41:0x0](ffff920a5087db88) 00000080:00200000:1.0:1788572250.268659:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41:0x0](ffff920a5087ec08) 00000080:00200000:1.0:1788572250.268666:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42:0x0](ffff920a5087aa08) 00000080:00200000:1.0:1788572250.268673:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42:0x0](ffff920a5087d348) 00000080:00200000:1.0:1788572250.268680:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43:0x0](ffff920a50873248) 00000080:00200000:1.0:1788572250.268688:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43:0x0](ffff920a50877448) 00000080:00200000:1.0:1788572250.268695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44:0x0](ffff920a508700c8) 00000080:00200000:1.0:1788572250.268703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44:0x0](ffff920a50875b88) 00000080:00200000:1.0:1788572250.268710:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45:0x0](ffff920a508442c8) 00000080:00200000:1.0:1788572250.268718:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45:0x0](ffff920a50842a08) 00000080:00200000:1.0:1788572250.268725:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46:0x0](ffff920a508721c8) 00000080:00200000:1.0:1788572250.268733:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46:0x0](ffff920a508763c8) 00000080:00200000:1.0:1788572250.268740:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x47:0x0](ffff920a5087cb08) 00000080:00200000:1.0:1788572250.268748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x47:0x0](ffff920a50879988) 00000080:00200000:1.0:1788572250.268756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x48:0x0](ffff920a5087a1c8) 00000080:00200000:1.0:1788572250.268763:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x48:0x0](ffff920a50879148) 00000080:00200000:1.0:1788572250.268770:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x49:0x0](ffff920a50871148) 00000080:00200000:1.0:1788572250.268777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x49:0x0](ffff920a50886c08) 00000080:00200000:1.0:1788572250.268785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4a:0x0](ffff920a5084f448) 00000080:00200000:1.0:1788572250.268793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4a:0x0](ffff920a5088ba88) 00000080:00200000:1.0:1788572250.268801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4b:0x0](ffff920a5087e3c8) 00000080:00200000:1.0:1788572250.268808:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4b:0x0](ffff920a508780c8) 00000080:00200000:1.0:1788572250.268815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4c:0x0](ffff920a5087f448) 00000080:00200000:1.0:1788572250.268822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4c:0x0](ffff920a5087b248) 00000080:00200000:1.0:1788572250.268829:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4d:0x0](ffff920a50845b88) 00000080:00200000:1.0:1788572250.268838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4d:0x0](ffff920a50846c08) 00000080:00200000:1.0:1788572250.268845:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4e:0x0](ffff920a5088c2c8) 00000080:00200000:1.0:1788572250.268853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4e:0x0](ffff920a5088aa08) 00000080:00200000:1.0:1788572250.268862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x4f:0x0](ffff920a508942c8) 00000080:00200000:1.0:1788572250.268872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x4f:0x0](ffff920a50892a08) 00000080:00200000:1.0:1788572250.268880:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x50:0x0](ffff920a50884b08) 00000080:00200000:1.0:1788572250.268889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x50:0x0](ffff920a50881988) 00000080:00200000:1.0:1788572250.268896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x51:0x0](ffff920a50896c08) 00000080:00200000:1.0:1788572250.268905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x51:0x0](ffff920a50895348) 00000080:00200000:1.0:1788572250.268912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x52:0x0](ffff920a508821c8) 00000080:00200000:1.0:1788572250.268925:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x52:0x0](ffff920a50881148) 00000080:00200000:1.0:1788572250.268945:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x53:0x0](ffff920a50888908) 00000080:00200000:1.0:1788572250.268955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x53:0x0](ffff920a5088db88) 00000080:00200000:1.0:1788572250.268963:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x54:0x0](ffff920a50840908) 00000080:00200000:1.0:1788572250.268972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x54:0x0](ffff920a508421c8) 00000080:00200000:1.0:1788572250.268980:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x55:0x0](ffff920a50880908) 00000080:00200000:1.0:1788572250.268987:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x55:0x0](ffff920a508842c8) 00000080:00200000:1.0:1788572250.268994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x56:0x0](ffff920a50841988) 00000080:00200000:1.0:1788572250.269001:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x56:0x0](ffff920a50844b08) 00000080:00200000:1.0:1788572250.269007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x57:0x0](ffff920a50897448) 00000080:00200000:1.0:1788572250.269014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x57:0x0](ffff920a508900c8) 00000080:00200000:1.0:1788572250.269021:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x58:0x0](ffff920a50894b08) 00000080:00200000:1.0:1788572250.269027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x58:0x0](ffff920a50891988) 00000080:00200000:1.0:1788572250.269035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x59:0x0](ffff920a50899148) 00000080:00200000:1.0:1788572250.269042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x59:0x0](ffff920a5089ba88) 00000080:00200000:1.0:1788572250.269050:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5a:0x0](ffff920a508921c8) 00000080:00200000:1.0:1788572250.269058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5a:0x0](ffff920a50891148) 00000080:00200000:1.0:1788572250.269065:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5b:0x0](ffff920a50893a88) 00000080:00200000:1.0:1788572250.269073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5b:0x0](ffff920a50890908) 00000080:00200000:1.0:1788572250.269080:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5c:0x0](ffff920a508963c8) 00000080:00200000:1.0:1788572250.269087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5c:0x0](ffff920a50893248) 00000080:00200000:1.0:1788572250.269095:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5d:0x0](ffff920a50895b88) 00000080:00200000:1.0:1788572250.269102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5d:0x0](ffff920a508a1148) 00000080:00200000:1.0:1788572250.269110:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5e:0x0](ffff920a50883a88) 00000080:00200000:1.0:1788572250.269118:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5e:0x0](ffff920a508863c8) 00000080:00200000:1.0:1788572250.269127:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x5f:0x0](ffff920a508a0908) 00000080:00200000:1.0:1788572250.269135:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x5f:0x0](ffff920a508a42c8) 00000080:00200000:1.0:1788572250.269143:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x60:0x0](ffff920a508a5b88) 00000080:00200000:1.0:1788572250.269151:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x60:0x0](ffff920a508a6c08) 00000080:00200000:1.0:1788572250.269158:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x61:0x0](ffff920a50887448) 00000080:00200000:1.0:1788572250.269166:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x61:0x0](ffff920a50883248) 00000080:00200000:1.0:1788572250.269174:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x62:0x0](ffff920a5088ec08) 00000080:00200000:1.0:1788572250.269183:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x62:0x0](ffff920a5088d348) 00000080:00200000:1.0:1788572250.269192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x63:0x0](ffff920a50898908) 00000080:00200000:1.0:1788572250.269201:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x63:0x0](ffff920a5089c2c8) 00000080:00200000:1.0:1788572250.269209:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x64:0x0](ffff920a5089aa08) 00000080:00200000:1.0:1788572250.269217:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x64:0x0](ffff920a5089db88) 00000080:00200000:1.0:1788572250.269225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x65:0x0](ffff920a50885348) 00000080:00200000:1.0:1788572250.269233:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x65:0x0](ffff920a508800c8) 00000080:00200000:1.0:1788572250.269240:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x66:0x0](ffff920a508a7448) 00000080:00200000:1.0:1788572250.269248:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x66:0x0](ffff920a508a00c8) 00000080:00200000:1.0:1788572250.269256:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x67:0x0](ffff920a50885b88) 00000080:00200000:1.0:1788572250.269264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x67:0x0](ffff920a50882a08) 00000080:00200000:1.0:1788572250.269271:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x68:0x0](ffff920a508a63c8) 00000080:00200000:1.0:1788572250.269278:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x68:0x0](ffff920a508a4b08) 00000080:00200000:1.0:1788572250.269285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x69:0x0](ffff920a508ab248) 00000080:00200000:1.0:1788572250.269293:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x69:0x0](ffff920a508af448) 00000080:00200000:1.0:1788572250.269300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6a:0x0](ffff920a508a80c8) 00000080:00200000:1.0:1788572250.269309:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6a:0x0](ffff920a508ae3c8) 00000080:00200000:1.0:1788572250.269318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6b:0x0](ffff920a5089d348) 00000080:00200000:1.0:1788572250.269325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6b:0x0](ffff920a5089b248) 00000080:00200000:1.0:1788572250.269332:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6c:0x0](ffff920a5089ec08) 00000080:00200000:1.0:1788572250.269341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6c:0x0](ffff920a5089f448) 00000080:00200000:1.0:1788572250.269349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6d:0x0](ffff920a508a21c8) 00000080:00200000:1.0:1788572250.269357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6d:0x0](ffff920a508a1988) 00000080:00200000:1.0:1788572250.269365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6e:0x0](ffff920a508a9988) 00000080:00200000:1.0:1788572250.269372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6e:0x0](ffff920a508aa1c8) 00000080:00200000:1.0:1788572250.269379:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x6f:0x0](ffff920a508a9148) 00000080:00200000:1.0:1788572250.269386:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x6f:0x0](ffff920a508aba88) 00000080:00200000:1.0:1788572250.269394:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x70:0x0](ffff920a5088e3c8) 00000080:00200000:1.0:1788572250.269402:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x70:0x0](ffff920a5088cb08) 00000080:00200000:1.0:1788572250.269411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x71:0x0](ffff920a508a2a08) 00000080:00200000:1.0:1788572250.269421:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x71:0x0](ffff920a508a3a88) 00000080:00200000:1.0:1788572250.269429:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x72:0x0](ffff920a508980c8) 00000080:00200000:1.0:1788572250.269438:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x72:0x0](ffff920a5089e3c8) 00000080:00200000:1.0:1788572250.269445:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x73:0x0](ffff920a508aaa08) 00000080:00200000:1.0:1788572250.269452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x73:0x0](ffff920a508adb88) 00000080:00200000:1.0:1788572250.269460:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x74:0x0](ffff920a5088a1c8) 00000080:00200000:1.0:1788572250.269469:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x74:0x0](ffff920a50889148) 00000080:00200000:1.0:1788572250.269477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x75:0x0](ffff920a508a3248) 00000080:00200000:1.0:1788572250.269485:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x75:0x0](ffff920a508a5348) 00000080:00200000:1.0:1788572250.269493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x76:0x0](ffff920a508b21c8) 00000080:00200000:1.0:1788572250.269500:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x76:0x0](ffff920a508b1148) 00000080:00200000:1.0:1788572250.269508:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x77:0x0](ffff920a508aec08) 00000080:00200000:1.0:1788572250.269515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x77:0x0](ffff920a508ac2c8) 00000080:00200000:1.0:1788572250.269522:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x78:0x0](ffff920a508b42c8) 00000080:00200000:1.0:1788572250.269529:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x78:0x0](ffff920a508b2a08) 00000080:00200000:1.0:1788572250.269536:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x79:0x0](ffff920a508b0908) 00000080:00200000:1.0:1788572250.269543:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x79:0x0](ffff920a508b5b88) 00000080:00200000:1.0:1788572250.269550:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7a:0x0](ffff920a508a8908) 00000080:00200000:1.0:1788572250.269556:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7a:0x0](ffff920a508acb08) 00000080:00200000:1.0:1788572250.269563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7b:0x0](ffff920a508b5348) 00000080:00200000:1.0:1788572250.269570:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7b:0x0](ffff920a508b3248) 00000080:00200000:1.0:1788572250.269577:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7c:0x0](ffff920a508ad348) 00000080:00200000:1.0:1788572250.269610:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7c:0x0](ffff920a508ba1c8) 00000080:00200000:1.0:1788572250.269619:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7d:0x0](ffff920a5088f448) 00000080:00200000:1.0:1788572250.269628:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7d:0x0](ffff920a5088b248) 00000080:00200000:1.0:1788572250.269636:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7e:0x0](ffff920a508880c8) 00000080:00200000:1.0:1788572250.269644:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7e:0x0](ffff920a50889988) 00000080:00200000:1.0:1788572250.269652:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x7f:0x0](ffff920a508c0908) 00000080:00200000:1.0:1788572250.269660:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x7f:0x0](ffff920a508c42c8) 00000080:00200000:1.0:1788572250.269668:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x80:0x0](ffff920a50899988) 00000080:00200000:1.0:1788572250.269676:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x80:0x0](ffff920a5089a1c8) 00000080:00200000:1.0:1788572250.269683:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x81:0x0](ffff920a508b8908) 00000080:00200000:1.0:1788572250.269691:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x81:0x0](ffff920a508bc2c8) 00000080:00200000:1.0:1788572250.269700:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x82:0x0](ffff920a508bba88) 00000080:00200000:1.0:1788572250.269708:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x82:0x0](ffff920a508baa08) 00000080:00200000:1.0:1788572250.269716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x83:0x0](ffff920a508bec08) 00000080:00200000:1.0:1788572250.269724:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x83:0x0](ffff920a508bd348) 00000080:00200000:1.0:1788572250.269732:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x84:0x0](ffff920a508bf448) 00000080:00200000:1.0:1788572250.269739:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x84:0x0](ffff920a508b80c8) 00000080:00200000:1.0:1788572250.269747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x85:0x0](ffff920a508bb248) 00000080:00200000:1.0:1788572250.269755:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x85:0x0](ffff920a508be3c8) 00000080:00200000:1.0:1788572250.269770:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x86:0x0](ffff920a508b4b08) 00000080:00200000:1.0:1788572250.269778:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x86:0x0](ffff920a508b1988) 00000080:00200000:1.0:1788572250.269787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x87:0x0](ffff920a508b63c8) 00000080:00200000:1.0:1788572250.269794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x87:0x0](ffff920a508b00c8) 00000080:00200000:1.0:1788572250.269801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x88:0x0](ffff920a508b7448) 00000080:00200000:1.0:1788572250.269809:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x88:0x0](ffff920a508b6c08) 00000080:00200000:1.0:1788572250.269818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x89:0x0](ffff920a508b3a88) 00000080:00200000:1.0:1788572250.269826:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x89:0x0](ffff920a508c80c8) 00000080:00200000:1.0:1788572250.269835:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8a:0x0](ffff920a508b9148) 00000080:00200000:1.0:1788572250.269844:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8a:0x0](ffff920a508bcb08) 00000080:00200000:1.0:1788572250.269851:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8b:0x0](ffff920a508ccb08) 00000080:00200000:1.0:1788572250.269859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8b:0x0](ffff920a508c9988) 00000080:00200000:1.0:1788572250.269867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8c:0x0](ffff920a508c5b88) 00000080:00200000:1.0:1788572250.269875:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8c:0x0](ffff920a508c6c08) 00000080:00200000:1.0:1788572250.269883:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8d:0x0](ffff920a508b9988) 00000080:00200000:1.0:1788572250.269891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8d:0x0](ffff920a508d21c8) 00000080:00200000:1.0:1788572250.269900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8e:0x0](ffff920a508ca1c8) 00000080:00200000:1.0:1788572250.269908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8e:0x0](ffff920a508d1148) 00000080:00200000:1.0:1788572250.269917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x8f:0x0](ffff920a508cba88) 00000080:00200000:1.0:1788572250.269940:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x8f:0x0](ffff920a508c8908) 00000080:00200000:1.0:1788572250.269951:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x90:0x0](ffff920a508c2a08) 00000080:00200000:1.0:1788572250.269963:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x90:0x0](ffff920a508c5348) 00000080:00200000:1.0:1788572250.269970:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x91:0x0](ffff920a508c3248) 00000080:00200000:1.0:1788572250.269978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x91:0x0](ffff920a508c7448) 00000080:00200000:1.0:1788572250.269987:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x92:0x0](ffff920a508d3a88) 00000080:00200000:1.0:1788572250.269995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x92:0x0](ffff920a508d0908) 00000080:00200000:1.0:1788572250.270003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x93:0x0](ffff920a508d42c8) 00000080:00200000:1.0:1788572250.270010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x93:0x0](ffff920a508d2a08) 00000080:00200000:1.0:1788572250.270018:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x94:0x0](ffff920a508cec08) 00000080:00200000:1.0:1788572250.270027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x94:0x0](ffff920a508cd348) 00000080:00200000:1.0:1788572250.270035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x95:0x0](ffff920a5089cb08) 00000080:00200000:1.0:1788572250.270045:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x95:0x0](ffff920a508da1c8) 00000080:00200000:1.0:1788572250.270053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x96:0x0](ffff920a508cb248) 00000080:00200000:1.0:1788572250.270061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x96:0x0](ffff920a508cf448) 00000080:00200000:1.0:1788572250.270069:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x97:0x0](ffff920a508c63c8) 00000080:00200000:1.0:1788572250.270078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x97:0x0](ffff920a508c4b08) 00000080:00200000:1.0:1788572250.270087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x98:0x0](ffff920a508dba88) 00000080:00200000:1.0:1788572250.270097:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x98:0x0](ffff920a508d8908) 00000080:00200000:1.0:1788572250.270106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x99:0x0](ffff920a508cdb88) 00000080:00200000:1.0:1788572250.270114:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x99:0x0](ffff920a508caa08) 00000080:00200000:1.0:1788572250.270122:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9a:0x0](ffff920a508cc2c8) 00000080:00200000:1.0:1788572250.270130:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9a:0x0](ffff920a508c9148) 00000080:00200000:1.0:1788572250.270138:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9b:0x0](ffff920a508d3248) 00000080:00200000:1.0:1788572250.270146:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9b:0x0](ffff920a508d7448) 00000080:00200000:1.0:1788572250.270154:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9c:0x0](ffff920a508ce3c8) 00000080:00200000:1.0:1788572250.270163:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9c:0x0](ffff920a508e63c8) 00000080:00200000:1.0:1788572250.270172:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9d:0x0](ffff920a508e4b08) 00000080:00200000:1.0:1788572250.270181:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9d:0x0](ffff920a508e1988) 00000080:00200000:1.0:1788572250.270189:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9e:0x0](ffff920a508dc2c8) 00000080:00200000:1.0:1788572250.270199:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9e:0x0](ffff920a508daa08) 00000080:00200000:1.0:1788572250.270207:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x9f:0x0](ffff920a508e1148) 00000080:00200000:1.0:1788572250.270219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x9f:0x0](ffff920a508e3a88) 00000080:00200000:1.0:1788572250.270228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa0:0x0](ffff920a508d1988) 00000080:00200000:1.0:1788572250.270237:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa0:0x0](ffff920a508e0908) 00000080:00200000:1.0:1788572250.270246:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa1:0x0](ffff920a508d4b08) 00000080:00200000:1.0:1788572250.270255:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa1:0x0](ffff920a508d63c8) 00000080:00200000:1.0:1788572250.270264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa2:0x0](ffff920a508e5b88) 00000080:00200000:1.0:1788572250.270278:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa2:0x0](ffff920a508e6c08) 00000080:00200000:1.0:1788572250.270287:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa3:0x0](ffff920a508e5348) 00000080:00200000:1.0:1788572250.270297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa3:0x0](ffff920a508e3248) 00000080:00200000:1.0:1788572250.270306:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa4:0x0](ffff920a508ddb88) 00000080:00200000:1.0:1788572250.270316:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa4:0x0](ffff920a508dec08) 00000080:00200000:1.0:1788572250.270324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa5:0x0](ffff920a508c1988) 00000080:00200000:1.0:1788572250.270334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa5:0x0](ffff920a508c21c8) 00000080:00200000:1.0:1788572250.270341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa6:0x0](ffff920a508d00c8) 00000080:00200000:1.0:1788572250.270349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa6:0x0](ffff920a508d5348) 00000080:00200000:1.0:1788572250.270357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa7:0x0](ffff920a508e00c8) 00000080:00200000:1.0:1788572250.270366:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa7:0x0](ffff920a508e7448) 00000080:00200000:1.0:1788572250.270374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa8:0x0](ffff920a508dd348) 00000080:00200000:1.0:1788572250.270382:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa8:0x0](ffff920a508db248) 00000080:00200000:1.0:1788572250.270390:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xa9:0x0](ffff920a508e42c8) 00000080:00200000:1.0:1788572250.270399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xa9:0x0](ffff920a508e21c8) 00000080:00200000:1.0:1788572250.270407:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xaa:0x0](ffff920a508d6c08) 00000080:00200000:1.0:1788572250.270415:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xaa:0x0](ffff920a508bdb88) 00000080:00200000:1.0:1788572250.270423:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xab:0x0](ffff920a508e2a08) 00000080:00200000:1.0:1788572250.270431:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xab:0x0](ffff920a508eba88) 00000080:00200000:1.0:1788572250.270439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xac:0x0](ffff920a508df448) 00000080:00200000:1.0:1788572250.270448:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xac:0x0](ffff920a508d80c8) 00000080:00200000:1.0:1788572250.270455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xad:0x0](ffff920a508d5b88) 00000080:00200000:1.0:1788572250.270464:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xad:0x0](ffff920a508f0908) 00000080:00200000:1.0:1788572250.270473:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xae:0x0](ffff920a508f2a08) 00000080:00200000:1.0:1788572250.270481:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xae:0x0](ffff920a508f5b88) 00000080:00200000:1.0:1788572250.270489:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xaf:0x0](ffff920a508f42c8) 00000080:00200000:1.0:1788572250.270497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xaf:0x0](ffff920a508f6c08) 00000080:00200000:1.0:1788572250.270506:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb0:0x0](ffff920a508f3248) 00000080:00200000:1.0:1788572250.270514:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb0:0x0](ffff920a508f7448) 00000080:00200000:1.0:1788572250.270523:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb1:0x0](ffff920a508c1148) 00000080:00200000:1.0:1788572250.270532:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb1:0x0](ffff920a508c3a88) 00000080:00200000:1.0:1788572250.270540:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb2:0x0](ffff920a508f00c8) 00000080:00200000:1.0:1788572250.270548:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb2:0x0](ffff920a508f63c8) 00000080:00200000:1.0:1788572250.270556:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb3:0x0](ffff920a508f5348) 00000080:00200000:1.0:1788572250.270565:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb3:0x0](ffff920a508f4b08) 00000080:00200000:1.0:1788572250.270576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb4:0x0](ffff920a508edb88) 00000080:00200000:1.0:1788572250.270609:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb4:0x0](ffff920a508eec08) 00000080:00200000:1.0:1788572250.270620:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb5:0x0](ffff920a508c00c8) 00000080:00200000:1.0:1788572250.270629:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb5:0x0](ffff920a508fc2c8) 00000080:00200000:1.0:1788572250.270636:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb6:0x0](ffff920a508ed348) 00000080:00200000:1.0:1788572250.270645:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb6:0x0](ffff920a508eb248) 00000080:00200000:1.0:1788572250.270654:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb7:0x0](ffff920a508ef448) 00000080:00200000:1.0:1788572250.270663:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb7:0x0](ffff920a508e80c8) 00000080:00200000:1.0:1788572250.270671:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb8:0x0](ffff920a508ecb08) 00000080:00200000:1.0:1788572250.270679:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb8:0x0](ffff920a508e9988) 00000080:00200000:1.0:1788572250.270687:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xb9:0x0](ffff920a508faa08) 00000080:00200000:1.0:1788572250.270695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xb9:0x0](ffff920a508fdb88) 00000080:00200000:1.0:1788572250.270703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xba:0x0](ffff920a508f1148) 00000080:00200000:1.0:1788572250.270711:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xba:0x0](ffff920a508f3a88) 00000080:00200000:1.0:1788572250.270719:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xbb:0x0](ffff920a508de3c8) 00000080:00200000:1.0:1788572250.270729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xbb:0x0](ffff920a508dcb08) 00000080:00200000:1.0:1788572250.270737:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xbc:0x0](ffff920a508fd348) 00000080:00200000:1.0:1788572250.270746:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xbc:0x0](ffff920a508fb248) 00000080:00200000:1.0:1788572250.270754:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xbd:0x0](ffff920a508f21c8) 00000080:00200000:1.0:1788572250.270763:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xbd:0x0](ffff920a50907448) 00000080:00200000:1.0:1788572250.270771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xbe:0x0](ffff920a508d9988) 00000080:00200000:1.0:1788572250.270781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xbe:0x0](ffff920a508d9148) 00000080:00200000:1.0:1788572250.270790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xbf:0x0](ffff920a508e9148) 00000080:00200000:1.0:1788572250.270799:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xbf:0x0](ffff920a508ea1c8) 00000080:00200000:1.0:1788572250.270807:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc0:0x0](ffff920a509063c8) 00000080:00200000:1.0:1788572250.270816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc0:0x0](ffff920a50904b08) 00000080:00200000:1.0:1788572250.270824:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc1:0x0](ffff920a508ec2c8) 00000080:00200000:1.0:1788572250.270832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc1:0x0](ffff920a508e8908) 00000080:00200000:1.0:1788572250.270840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc2:0x0](ffff920a50901988) 00000080:00200000:1.0:1788572250.270848:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc2:0x0](ffff920a509021c8) 00000080:00200000:1.0:1788572250.270856:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc3:0x0](ffff920a50901148) 00000080:00200000:1.0:1788572250.270864:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc3:0x0](ffff920a50903a88) 00000080:00200000:1.0:1788572250.270872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc4:0x0](ffff920a50900908) 00000080:00200000:1.0:1788572250.270879:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc4:0x0](ffff920a509042c8) 00000080:00200000:1.0:1788572250.270885:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc5:0x0](ffff920a508ee3c8) 00000080:00200000:1.0:1788572250.270893:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc5:0x0](ffff920a508eaa08) 00000080:00200000:1.0:1788572250.270901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc6:0x0](ffff920a5090ba88) 00000080:00200000:1.0:1788572250.270908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc6:0x0](ffff920a50908908) 00000080:00200000:1.0:1788572250.270916:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc7:0x0](ffff920a5090c2c8) 00000080:00200000:1.0:1788572250.270924:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc7:0x0](ffff920a5090aa08) 00000080:00200000:1.0:1788572250.273752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc8:0x0](ffff920a50913a88) 00000080:00200000:1.0:1788572250.273785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc8:0x0](ffff920a50910908) 00000080:00200000:1.0:1788572250.273796:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xc9:0x0](ffff920a5090db88) 00000080:00200000:1.0:1788572250.273806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xc9:0x0](ffff920a5090ec08) 00000080:00200000:1.0:1788572250.273814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xca:0x0](ffff920a50905348) 00000080:00200000:1.0:1788572250.273825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xca:0x0](ffff920a50906c08) 00000080:00200000:1.0:1788572250.273839:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xcb:0x0](ffff920a5090b248) 00000080:00200000:1.0:1788572250.273849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xcb:0x0](ffff920a5090f448) 00000080:00200000:1.0:1788572250.273857:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xcc:0x0](ffff920a508f80c8) 00000080:00200000:1.0:1788572250.273872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xcc:0x0](ffff920a508fe3c8) 00000080:00200000:1.0:1788572250.273881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xcd:0x0](ffff920a508f9988) 00000080:00200000:1.0:1788572250.273889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xcd:0x0](ffff920a508fa1c8) 00000080:00200000:1.0:1788572250.273899:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xce:0x0](ffff920a50905b88) 00000080:00200000:1.0:1788572250.273908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xce:0x0](ffff920a50902a08) 00000080:00200000:1.0:1788572250.273917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xcf:0x0](ffff920a5090cb08) 00000080:00200000:1.0:1788572250.273941:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xcf:0x0](ffff920a50909988) 00000080:00200000:1.0:1788572250.273955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd0:0x0](ffff920a50903248) 00000080:00200000:1.0:1788572250.273964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd0:0x0](ffff920a508f1988) 00000080:00200000:1.0:1788572250.273972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd1:0x0](ffff920a509142c8) 00000080:00200000:1.0:1788572250.273982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd1:0x0](ffff920a50912a08) 00000080:00200000:1.0:1788572250.273992:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd2:0x0](ffff920a5090a1c8) 00000080:00200000:1.0:1788572250.274000:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd2:0x0](ffff920a50909148) 00000080:00200000:1.0:1788572250.274009:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd3:0x0](ffff920a5090e3c8) 00000080:00200000:1.0:1788572250.274018:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd3:0x0](ffff920a509080c8) 00000080:00200000:1.0:1788572250.274027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd4:0x0](ffff920a5091aa08) 00000080:00200000:1.0:1788572250.274036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd4:0x0](ffff920a5091db88) 00000080:00200000:1.0:1788572250.274044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd5:0x0](ffff920a50916c08) 00000080:00200000:1.0:1788572250.274058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd5:0x0](ffff920a50915348) 00000080:00200000:1.0:1788572250.274073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd6:0x0](ffff920a5091d348) 00000080:00200000:1.0:1788572250.274082:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd6:0x0](ffff920a5091b248) 00000080:00200000:1.0:1788572250.274090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd7:0x0](ffff920a5090d348) 00000080:00200000:1.0:1788572250.274098:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd7:0x0](ffff920a509000c8) 00000080:00200000:1.0:1788572250.274107:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd8:0x0](ffff920a509180c8) 00000080:00200000:1.0:1788572250.275609:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd8:0x0](ffff920a5091e3c8) 00000080:00200000:1.0:1788572250.275642:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xd9:0x0](ffff920a50927448) 00000080:00200000:1.0:1788572250.275654:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xd9:0x0](ffff920a509200c8) 00000080:00200000:1.0:1788572250.275663:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xda:0x0](ffff920a50919988) 00000080:00200000:1.0:1788572250.275671:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xda:0x0](ffff920a5091a1c8) 00000080:00200000:1.0:1788572250.275682:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xdb:0x0](ffff920a509263c8) 00000080:00200000:1.0:1788572250.275692:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xdb:0x0](ffff920a50924b08) 00000080:00200000:1.0:1788572250.275701:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xdc:0x0](ffff920a50913248) 00000080:00200000:1.0:1788572250.275717:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xdc:0x0](ffff920a50917448) 00000080:00200000:1.0:1788572250.275728:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xdd:0x0](ffff920a50919148) 00000080:00200000:1.0:1788572250.275738:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xdd:0x0](ffff920a5091ba88) 00000080:00200000:1.0:1788572250.275747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xde:0x0](ffff920a509100c8) 00000080:00200000:1.0:1788572250.275758:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xde:0x0](ffff920a509163c8) 00000080:00200000:1.0:1788572250.275767:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xdf:0x0](ffff920a50920908) 00000080:00200000:1.0:1788572250.275775:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xdf:0x0](ffff920a509242c8) 00000080:00200000:1.0:1788572250.275785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe0:0x0](ffff920a5091f448) 00000080:00200000:1.0:1788572250.275794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe0:0x0](ffff920a5091ec08) 00000080:00200000:1.0:1788572250.275803:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe1:0x0](ffff920a50922a08) 00000080:00200000:1.0:1788572250.275812:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe1:0x0](ffff920a50925b88) 00000080:00200000:1.0:1788572250.275821:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe2:0x0](ffff920a50914b08) 00000080:00200000:1.0:1788572250.275831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe2:0x0](ffff920a50911988) 00000080:00200000:1.0:1788572250.275842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe3:0x0](ffff920a50918908) 00000080:00200000:1.0:1788572250.275850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe3:0x0](ffff920a5091c2c8) 00000080:00200000:1.0:1788572250.275860:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe4:0x0](ffff920a5092aa08) 00000080:00200000:1.0:1788572250.275868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe4:0x0](ffff920a5092db88) 00000080:00200000:1.0:1788572250.275877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe5:0x0](ffff920a5092ec08) 00000080:00200000:1.0:1788572250.275886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe5:0x0](ffff920a5092d348) 00000080:00200000:1.0:1788572250.275894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe6:0x0](ffff920a50925348) 00000080:00200000:1.0:1788572250.275903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe6:0x0](ffff920a50926c08) 00000080:00200000:1.0:1788572250.275913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe7:0x0](ffff920a50923a88) 00000080:00200000:1.0:1788572250.275922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe7:0x0](ffff920a50921148) 00000080:00200000:1.0:1788572250.275946:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe8:0x0](ffff920a509280c8) 00000080:00200000:1.0:1788572250.275956:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe8:0x0](ffff920a5092e3c8) 00000080:00200000:1.0:1788572250.275965:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xe9:0x0](ffff920a50929988) 00000080:00200000:1.0:1788572250.275982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xe9:0x0](ffff920a5092a1c8) 00000080:00200000:1.0:1788572250.275991:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xea:0x0](ffff920a508fba88) 00000080:00200000:1.0:1788572250.276007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xea:0x0](ffff920a508f9148) 00000080:00200000:1.0:1788572250.276019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xeb:0x0](ffff920a5092cb08) 00000080:00200000:1.0:1788572250.276028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xeb:0x0](ffff920a50929148) 00000080:00200000:1.0:1788572250.276036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xec:0x0](ffff920a509221c8) 00000080:00200000:1.0:1788572250.276045:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xec:0x0](ffff920a50921988) 00000080:00200000:1.0:1788572250.276052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xed:0x0](ffff920a50928908) 00000080:00200000:1.0:1788572250.276061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xed:0x0](ffff920a5092c2c8) 00000080:00200000:1.0:1788572250.276070:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xee:0x0](ffff920a50936c08) 00000080:00200000:1.0:1788572250.276079:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xee:0x0](ffff920a50935348) 00000080:00200000:1.0:1788572250.276089:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xef:0x0](ffff920a5092ba88) 00000080:00200000:1.0:1788572250.276096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xef:0x0](ffff920a5092f448) 00000080:00200000:1.0:1788572250.276106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf0:0x0](ffff920a5092b248) 00000080:00200000:1.0:1788572250.277239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf0:0x0](ffff920a5091cb08) 00000080:00200000:1.0:1788572250.277275:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf1:0x0](ffff920a509300c8) 00000080:00200000:1.0:1788572250.277287:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf1:0x0](ffff920a509363c8) 00000080:00200000:1.0:1788572250.277297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf2:0x0](ffff920a50911148) 00000080:00200000:1.0:1788572250.277307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf2:0x0](ffff920a509121c8) 00000080:00200000:1.0:1788572250.277315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf3:0x0](ffff920a509321c8) 00000080:00200000:1.0:1788572250.277324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf3:0x0](ffff920a50931148) 00000080:00200000:1.0:1788572250.277333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf4:0x0](ffff920a508fcb08) 00000080:00200000:1.0:1788572250.277343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf4:0x0](ffff920a508ff448) 00000080:00200000:1.0:1788572250.277352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf5:0x0](ffff920a50915b88) 00000080:00200000:1.0:1788572250.277360:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf5:0x0](ffff920a50939988) 00000080:00200000:1.0:1788572250.277369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf6:0x0](ffff920a50945348) 00000080:00200000:1.0:1788572250.277378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf6:0x0](ffff920a50943248) 00000080:00200000:1.0:1788572250.277387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf7:0x0](ffff920a50931988) 00000080:00200000:1.0:1788572250.277396:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf7:0x0](ffff920a50933a88) 00000080:00200000:1.0:1788572250.277405:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf8:0x0](ffff920a508f8908) 00000080:00200000:1.0:1788572250.277423:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf8:0x0](ffff920a508fec08) 00000080:00200000:1.0:1788572250.277432:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xf9:0x0](ffff920a509463c8) 00000080:00200000:1.0:1788572250.277441:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xf9:0x0](ffff920a50944b08) 00000080:00200000:1.0:1788572250.277450:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xfa:0x0](ffff920a509342c8) 00000080:00200000:1.0:1788572250.277580:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xfa:0x0](ffff920a50932a08) 00000080:00200000:1.0:1788572250.277608:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xfb:0x0](ffff920a50930908) 00000080:00200000:1.0:1788572250.277618:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xfb:0x0](ffff920a50935b88) 00000080:00200000:1.0:1788572250.277627:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xfc:0x0](ffff920a5094f448) 00000080:00200000:1.0:1788572250.277638:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xfc:0x0](ffff920a509480c8) 00000080:00200000:1.0:1788572250.277648:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xfd:0x0](ffff920a50934b08) 00000080:00200000:1.0:1788572250.277657:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xfd:0x0](ffff920a50937448) 00000080:00200000:1.0:1788572250.277673:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xfe:0x0](ffff920a5093a1c8) 00000080:00200000:1.0:1788572250.277684:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xfe:0x0](ffff920a50939148) 00000080:00200000:1.0:1788572250.277694:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0xff:0x0](ffff920a50923248) 00000080:00200000:1.0:1788572250.277703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0xff:0x0](ffff920a50951988) 00000080:00200000:1.0:1788572250.277712:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x100:0x0](ffff920a509421c8) 00000080:00200000:1.0:1788572250.277727:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x100:0x0](ffff920a50941148) 00000080:00200000:1.0:1788572250.277736:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x101:0x0](ffff920a50943a88) 00000080:00200000:1.0:1788572250.277745:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x101:0x0](ffff920a50940908) 00000080:00200000:1.0:1788572250.277753:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x102:0x0](ffff920a509442c8) 00000080:00200000:1.0:1788572250.277762:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x102:0x0](ffff920a50942a08) 00000080:00200000:1.0:1788572250.277770:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x103:0x0](ffff920a50949988) 00000080:00200000:1.0:1788572250.277780:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x103:0x0](ffff920a5094a1c8) 00000080:00200000:1.0:1788572250.277789:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x104:0x0](ffff920a50946c08) 00000080:00200000:1.0:1788572250.277798:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x104:0x0](ffff920a50945b88) 00000080:00200000:1.0:1788572250.277808:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x105:0x0](ffff920a50953a88) 00000080:00200000:1.0:1788572250.277816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x105:0x0](ffff920a50950908) 00000080:00200000:1.0:1788572250.277825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x106:0x0](ffff920a509400c8) 00000080:00200000:1.0:1788572250.277834:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x106:0x0](ffff920a50947448) 00000080:00200000:1.0:1788572250.277843:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x107:0x0](ffff920a50941988) 00000080:00200000:1.0:1788572250.277850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x107:0x0](ffff920a5095f448) 00000080:00200000:1.0:1788572250.277858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x108:0x0](ffff920a509580c8) 00000080:00200000:1.0:1788572250.277866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x108:0x0](ffff920a5095e3c8) 00000080:00200000:1.0:1788572250.277873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x109:0x0](ffff920a50959988) 00000080:00200000:1.0:1788572250.277882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x109:0x0](ffff920a5095a1c8) 00000080:00200000:1.0:1788572250.277890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10a:0x0](ffff920a5094ba88) 00000080:00200000:1.0:1788572250.277900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10a:0x0](ffff920a50948908) 00000080:00200000:1.0:1788572250.277908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10b:0x0](ffff920a50955b88) 00000080:00200000:1.0:1788572250.277917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10b:0x0](ffff920a50956c08) 00000080:00200000:1.0:1788572250.277926:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10c:0x0](ffff920a50938908) 00000080:00200000:1.0:1788572250.277949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10c:0x0](ffff920a5093c2c8) 00000080:00200000:1.0:1788572250.277958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10d:0x0](ffff920a5094c2c8) 00000080:00200000:1.0:1788572250.277966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10d:0x0](ffff920a5094aa08) 00000080:00200000:1.0:1788572250.277974:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10e:0x0](ffff920a5094db88) 00000080:00200000:1.0:1788572250.277982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10e:0x0](ffff920a5094ec08) 00000080:00200000:1.0:1788572250.277998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x10f:0x0](ffff920a5093aa08) 00000080:00200000:1.0:1788572250.278008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x10f:0x0](ffff920a5093db88) 00000080:00200000:1.0:1788572250.278018:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x110:0x0](ffff920a50957448) 00000080:00200000:1.0:1788572250.278026:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x110:0x0](ffff920a509500c8) 00000080:00200000:1.0:1788572250.278034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x111:0x0](ffff920a50958908) 00000080:00200000:1.0:1788572250.278042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x111:0x0](ffff920a5095c2c8) 00000080:00200000:1.0:1788572250.278050:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x112:0x0](ffff920a5095db88) 00000080:00200000:1.0:1788572250.278058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x112:0x0](ffff920a5095ec08) 00000080:00200000:1.0:1788572250.278065:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x113:0x0](ffff920a50954b08) 00000080:00200000:1.0:1788572250.278073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x113:0x0](ffff920a509563c8) 00000080:00200000:1.0:1788572250.278081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x114:0x0](ffff920a5095b248) 00000080:00200000:1.0:1788572250.278088:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x114:0x0](ffff920a5095d348) 00000080:00200000:1.0:1788572250.278096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x115:0x0](ffff920a5093ec08) 00000080:00200000:1.0:1788572250.278104:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x115:0x0](ffff920a5093d348) 00000080:00200000:1.0:1788572250.278112:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x116:0x0](ffff920a5095ba88) 00000080:00200000:1.0:1788572250.279234:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x116:0x0](ffff920a50959148) 00000080:00200000:1.0:1788572250.279247:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x117:0x0](ffff920a5093f448) 00000080:00200000:1.0:1788572250.279262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x117:0x0](ffff920a509380c8) 00000080:00200000:1.0:1788572250.279272:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x118:0x0](ffff920a5095cb08) 00000080:00200000:1.0:1788572250.279281:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x118:0x0](ffff920a5095aa08) 00000080:00200000:1.0:1788572250.279291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x119:0x0](ffff920a50963a88) 00000080:00200000:1.0:1788572250.279301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x119:0x0](ffff920a50960908) 00000080:00200000:1.0:1788572250.279309:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11a:0x0](ffff920a509542c8) 00000080:00200000:1.0:1788572250.279318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11a:0x0](ffff920a50951148) 00000080:00200000:1.0:1788572250.279327:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11b:0x0](ffff920a509521c8) 00000080:00200000:1.0:1788572250.279335:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11b:0x0](ffff920a50952a08) 00000080:00200000:1.0:1788572250.279342:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11c:0x0](ffff920a50962a08) 00000080:00200000:1.0:1788572250.279352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11c:0x0](ffff920a50965b88) 00000080:00200000:1.0:1788572250.279361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11d:0x0](ffff920a50953248) 00000080:00200000:1.0:1788572250.279370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11d:0x0](ffff920a50955348) 00000080:00200000:1.0:1788572250.279378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11e:0x0](ffff920a50963248) 00000080:00200000:1.0:1788572250.279387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11e:0x0](ffff920a50967448) 00000080:00200000:1.0:1788572250.279395:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x11f:0x0](ffff920a50933248) 00000080:00200000:1.0:1788572250.279403:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x11f:0x0](ffff920a50969988) 00000080:00200000:1.0:1788572250.279411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x120:0x0](ffff920a50965348) 00000080:00200000:1.0:1788572250.279420:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x120:0x0](ffff920a509600c8) 00000080:00200000:1.0:1788572250.279428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x121:0x0](ffff920a5096ba88) 00000080:00200000:1.0:1788572250.279436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x121:0x0](ffff920a50968908) 00000080:00200000:1.0:1788572250.279444:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x122:0x0](ffff920a509663c8) 00000080:00200000:1.0:1788572250.279453:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x122:0x0](ffff920a50964b08) 00000080:00200000:1.0:1788572250.280189:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x123:0x0](ffff920a5096aa08) 00000080:00200000:1.0:1788572250.280209:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x123:0x0](ffff920a5096db88) 00000080:00200000:1.0:1788572250.280216:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x124:0x0](ffff920a50961988) 00000080:00200000:1.0:1788572250.280224:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x124:0x0](ffff920a509621c8) 00000080:00200000:1.0:1788572250.280233:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x125:0x0](ffff920a5093e3c8) 00000080:00200000:1.0:1788572250.280243:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x125:0x0](ffff920a5093cb08) 00000080:00200000:1.0:1788572250.280251:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x126:0x0](ffff920a5094d348) 00000080:00200000:1.0:1788572250.280262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x126:0x0](ffff920a50949148) 00000080:00200000:1.0:1788572250.280271:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x127:0x0](ffff920a5096d348) 00000080:00200000:1.0:1788572250.280280:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x127:0x0](ffff920a5096b248) 00000080:00200000:1.0:1788572250.280294:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x128:0x0](ffff920a5094cb08) 00000080:00200000:1.0:1788572250.280305:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x128:0x0](ffff920a5094e3c8) 00000080:00200000:1.0:1788572250.280314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x129:0x0](ffff920a5096f448) 00000080:00200000:1.0:1788572250.280322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x129:0x0](ffff920a509680c8) 00000080:00200000:1.0:1788572250.280330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12a:0x0](ffff920a50961148) 00000080:00200000:1.0:1788572250.280340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12a:0x0](ffff920a509642c8) 00000080:00200000:1.0:1788572250.280348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12b:0x0](ffff920a5096e3c8) 00000080:00200000:1.0:1788572250.280357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12b:0x0](ffff920a5096cb08) 00000080:00200000:1.0:1788572250.280365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12c:0x0](ffff920a5093ba88) 00000080:00200000:1.0:1788572250.280374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12c:0x0](ffff920a5093b248) 00000080:00200000:1.0:1788572250.280382:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12d:0x0](ffff920a5096ec08) 00000080:00200000:1.0:1788572250.280391:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12d:0x0](ffff920a5096c2c8) 00000080:00200000:1.0:1788572250.280399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12e:0x0](ffff920a5094b248) 00000080:00200000:1.0:1788572250.280408:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12e:0x0](ffff920a50977448) 00000080:00200000:1.0:1788572250.280417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x12f:0x0](ffff920a50969148) 00000080:00200000:1.0:1788572250.280425:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x12f:0x0](ffff920a5096a1c8) 00000080:00200000:1.0:1788572250.280433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x130:0x0](ffff920a50979148) 00000080:00200000:1.0:1788572250.280442:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x130:0x0](ffff920a5097ba88) 00000080:00200000:1.0:1788572250.280451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x131:0x0](ffff920a5097c2c8) 00000080:00200000:1.0:1788572250.280472:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x131:0x0](ffff920a5097aa08) 00000080:00200000:1.0:1788572250.280484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x132:0x0](ffff920a50966c08) 00000080:00200000:1.0:1788572250.280492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x132:0x0](ffff920a50986c08) 00000080:00200000:1.0:1788572250.280501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x133:0x0](ffff920a5098a1c8) 00000080:00200000:1.0:1788572250.280510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x133:0x0](ffff920a50989148) 00000080:00200000:1.0:1788572250.280520:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x134:0x0](ffff920a509700c8) 00000080:00200000:1.0:1788572250.280529:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x134:0x0](ffff920a509763c8) 00000080:00200000:1.0:1788572250.280539:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x135:0x0](ffff920a5097ec08) 00000080:00200000:1.0:1788572250.280550:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x135:0x0](ffff920a5097d348) 00000080:00200000:1.0:1788572250.280560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x136:0x0](ffff920a5098ba88) 00000080:00200000:1.0:1788572250.280569:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x136:0x0](ffff920a50988908) 00000080:00200000:1.0:1788572250.280578:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x137:0x0](ffff920a5098aa08) 00000080:00200000:1.0:1788572250.280613:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x137:0x0](ffff920a5098db88) 00000080:00200000:1.0:1788572250.280624:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x138:0x0](ffff920a509800c8) 00000080:00200000:1.0:1788572250.280633:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x138:0x0](ffff920a509863c8) 00000080:00200000:1.0:1788572250.280642:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x139:0x0](ffff920a5097f448) 00000080:00200000:1.0:1788572250.280652:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x139:0x0](ffff920a509780c8) 00000080:00200000:1.0:1788572250.280661:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13a:0x0](ffff920a50974b08) 00000080:00200000:1.0:1788572250.280669:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13a:0x0](ffff920a50971988) 00000080:00200000:1.0:1788572250.280677:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13b:0x0](ffff920a5097b248) 00000080:00200000:1.0:1788572250.280686:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13b:0x0](ffff920a5097e3c8) 00000080:00200000:1.0:1788572250.280694:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13c:0x0](ffff920a5098ec08) 00000080:00200000:1.0:1788572250.280703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13c:0x0](ffff920a5098d348) 00000080:00200000:1.0:1788572250.280711:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13d:0x0](ffff920a50979988) 00000080:00200000:1.0:1788572250.280721:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13d:0x0](ffff920a5097a1c8) 00000080:00200000:1.0:1788572250.280729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13e:0x0](ffff920a5097cb08) 00000080:00200000:1.0:1788572250.280737:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13e:0x0](ffff920a5097db88) 00000080:00200000:1.0:1788572250.280744:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x13f:0x0](ffff920a50978908) 00000080:00200000:1.0:1788572250.280753:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x13f:0x0](ffff920a50994b08) 00000080:00200000:1.0:1788572250.280761:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x140:0x0](ffff920a509921c8) 00000080:00200000:1.0:1788572250.280769:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x140:0x0](ffff920a50991148) 00000080:00200000:1.0:1788572250.280777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x141:0x0](ffff920a50981148) 00000080:00200000:1.0:1788572250.280787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x141:0x0](ffff920a50983a88) 00000080:00200000:1.0:1788572250.280795:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x142:0x0](ffff920a509842c8) 00000080:00200000:1.0:1788572250.280803:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x142:0x0](ffff920a50982a08) 00000080:00200000:1.0:1788572250.280812:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x143:0x0](ffff920a50993a88) 00000080:00200000:1.0:1788572250.280820:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x143:0x0](ffff920a50990908) 00000080:00200000:1.0:1788572250.280828:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x144:0x0](ffff920a509721c8) 00000080:00200000:1.0:1788572250.280837:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x144:0x0](ffff920a50971148) 00000080:00200000:1.0:1788572250.280845:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x145:0x0](ffff920a50985b88) 00000080:00200000:1.0:1788572250.280854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x145:0x0](ffff920a50980908) 00000080:00200000:1.0:1788572250.280863:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x146:0x0](ffff920a50995b88) 00000080:00200000:1.0:1788572250.280871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x146:0x0](ffff920a50996c08) 00000080:00200000:1.0:1788572250.280880:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x147:0x0](ffff920a50995348) 00000080:00200000:1.0:1788572250.280889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x147:0x0](ffff920a50993248) 00000080:00200000:1.0:1788572250.280897:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x148:0x0](ffff920a509821c8) 00000080:00200000:1.0:1788572250.280906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x148:0x0](ffff920a50981988) 00000080:00200000:1.0:1788572250.280915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x149:0x0](ffff920a50997448) 00000080:00200000:1.0:1788572250.280923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x149:0x0](ffff920a509900c8) 00000080:00200000:1.0:1788572250.282867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14a:0x0](ffff920a509963c8) 00000080:00200000:1.0:1788572250.282891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14a:0x0](ffff920a50992a08) 00000080:00200000:1.0:1788572250.282901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14b:0x0](ffff920a50987448) 00000080:00200000:1.0:1788572250.282913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14b:0x0](ffff920a50983248) 00000080:00200000:1.0:1788572250.282922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14c:0x0](ffff920a50985348) 00000080:00200000:1.0:1788572250.282944:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14c:0x0](ffff920a50984b08) 00000080:00200000:1.0:1788572250.282953:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14d:0x0](ffff920a5099b248) 00000080:00200000:1.0:1788572250.282961:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14d:0x0](ffff920a5099f448) 00000080:00200000:1.0:1788572250.282969:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14e:0x0](ffff920a509980c8) 00000080:00200000:1.0:1788572250.282977:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14e:0x0](ffff920a5099e3c8) 00000080:00200000:1.0:1788572250.282988:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x14f:0x0](ffff920a5098e3c8) 00000080:00200000:1.0:1788572250.283001:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x14f:0x0](ffff920a5098cb08) 00000080:00200000:1.0:1788572250.283010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x150:0x0](ffff920a509942c8) 00000080:00200000:1.0:1788572250.283018:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x150:0x0](ffff920a50991988) 00000080:00200000:1.0:1788572250.283026:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x151:0x0](ffff920a50989988) 00000080:00200000:1.0:1788572250.283035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x151:0x0](ffff920a509880c8) 00000080:00200000:1.0:1788572250.283044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x152:0x0](ffff920a5099cb08) 00000080:00200000:1.0:1788572250.283052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x152:0x0](ffff920a50999988) 00000080:00200000:1.0:1788572250.283059:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x153:0x0](ffff920a509a1148) 00000080:00200000:1.0:1788572250.283066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x153:0x0](ffff920a509a3a88) 00000080:00200000:1.0:1788572250.283075:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x154:0x0](ffff920a5099ba88) 00000080:00200000:1.0:1788572250.283082:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x154:0x0](ffff920a50998908) 00000080:00200000:1.0:1788572250.283090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x155:0x0](ffff920a5098f448) 00000080:00200000:1.0:1788572250.283098:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x155:0x0](ffff920a5098b248) 00000080:00200000:1.0:1788572250.283106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x156:0x0](ffff920a50999148) 00000080:00200000:1.0:1788572250.283221:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x156:0x0](ffff920a5099c2c8) 00000080:00200000:1.0:1788572250.283236:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x157:0x0](ffff920a50973a88) 00000080:00200000:1.0:1788572250.283245:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x157:0x0](ffff920a50970908) 00000080:00200000:1.0:1788572250.283254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x158:0x0](ffff920a5099aa08) 00000080:00200000:1.0:1788572250.283261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x158:0x0](ffff920a5099db88) 00000080:00200000:1.0:1788572250.283269:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x159:0x0](ffff920a509a5b88) 00000080:00200000:1.0:1788572250.283278:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x159:0x0](ffff920a509a6c08) 00000080:00200000:1.0:1788572250.283285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15a:0x0](ffff920a50972a08) 00000080:00200000:1.0:1788572250.283292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15a:0x0](ffff920a50975b88) 00000080:00200000:1.0:1788572250.283300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15b:0x0](ffff920a5099d348) 00000080:00200000:1.0:1788572250.283307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15b:0x0](ffff920a5099ec08) 00000080:00200000:1.0:1788572250.283314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15c:0x0](ffff920a509aec08) 00000080:00200000:1.0:1788572250.283322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15c:0x0](ffff920a509ad348) 00000080:00200000:1.0:1788572250.283331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15d:0x0](ffff920a509b21c8) 00000080:00200000:1.0:1788572250.283339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15d:0x0](ffff920a509b1148) 00000080:00200000:1.0:1788572250.283346:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15e:0x0](ffff920a509b3a88) 00000080:00200000:1.0:1788572250.283354:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15e:0x0](ffff920a509b0908) 00000080:00200000:1.0:1788572250.283363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x15f:0x0](ffff920a509a5348) 00000080:00200000:1.0:1788572250.283370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x15f:0x0](ffff920a509a3248) 00000080:00200000:1.0:1788572250.283378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x160:0x0](ffff920a509b2a08) 00000080:00200000:1.0:1788572250.283385:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x160:0x0](ffff920a509b5b88) 00000080:00200000:1.0:1788572250.283393:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x161:0x0](ffff920a509a63c8) 00000080:00200000:1.0:1788572250.283401:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x161:0x0](ffff920a509a4b08) 00000080:00200000:1.0:1788572250.283409:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x162:0x0](ffff920a50975348) 00000080:00200000:1.0:1788572250.283417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x162:0x0](ffff920a50973248) 00000080:00200000:1.0:1788572250.283425:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x163:0x0](ffff920a509ab248) 00000080:00200000:1.0:1788572250.283433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x163:0x0](ffff920a509af448) 00000080:00200000:1.0:1788572250.283441:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x164:0x0](ffff920a509a00c8) 00000080:00200000:1.0:1788572250.283448:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x164:0x0](ffff920a509a7448) 00000080:00200000:1.0:1788572250.283454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x165:0x0](ffff920a509a80c8) 00000080:00200000:1.0:1788572250.283664:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x165:0x0](ffff920a509ae3c8) 00000080:00200000:1.0:1788572250.283678:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x166:0x0](ffff920a509a0908) 00000080:00200000:1.0:1788572250.283687:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x166:0x0](ffff920a509a21c8) 00000080:00200000:1.0:1788572250.283696:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x167:0x0](ffff920a509acb08) 00000080:00200000:1.0:1788572250.283709:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x167:0x0](ffff920a509a9988) 00000080:00200000:1.0:1788572250.283716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x168:0x0](ffff920a509aa1c8) 00000080:00200000:1.0:1788572250.283725:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x168:0x0](ffff920a509a9148) 00000080:00200000:1.0:1788572250.283732:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x169:0x0](ffff920a509b00c8) 00000080:00200000:1.0:1788572250.283740:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x169:0x0](ffff920a509b63c8) 00000080:00200000:1.0:1788572250.283748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16a:0x0](ffff920a509a1988) 00000080:00200000:1.0:1788572250.283756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16a:0x0](ffff920a509a2a08) 00000080:00200000:1.0:1788572250.283764:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16b:0x0](ffff920a509aba88) 00000080:00200000:1.0:1788572250.283778:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16b:0x0](ffff920a509a8908) 00000080:00200000:1.0:1788572250.283786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16c:0x0](ffff920a509b1988) 00000080:00200000:1.0:1788572250.283794:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16c:0x0](ffff920a509b4b08) 00000080:00200000:1.0:1788572250.283802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16d:0x0](ffff920a509b3248) 00000080:00200000:1.0:1788572250.283809:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16d:0x0](ffff920a509b5348) 00000080:00200000:1.0:1788572250.283816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16e:0x0](ffff920a509b6c08) 00000080:00200000:1.0:1788572250.283823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16e:0x0](ffff920a509b42c8) 00000080:00200000:1.0:1788572250.283831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x16f:0x0](ffff920a509ac2c8) 00000080:00200000:1.0:1788572250.283840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x16f:0x0](ffff920a509aaa08) 00000080:00200000:1.0:1788572250.283847:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x170:0x0](ffff920a5099a1c8) 00000080:00200000:1.0:1788572250.283854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x170:0x0](ffff920a509bba88) 00000080:00200000:1.0:1788572250.283861:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x171:0x0](ffff920a509742c8) 00000080:00200000:1.0:1788572250.283871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x171:0x0](ffff920a50976c08) 00000080:00200000:1.0:1788572250.283878:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x172:0x0](ffff920a509c42c8) 00000080:00200000:1.0:1788572250.283886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x172:0x0](ffff920a509c2a08) 00000080:00200000:1.0:1788572250.283895:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x173:0x0](ffff920a509c6c08) 00000080:00200000:1.0:1788572250.283903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x173:0x0](ffff920a509c5348) 00000080:00200000:1.0:1788572250.283912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x174:0x0](ffff920a509c7448) 00000080:00200000:1.0:1788572250.283920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x174:0x0](ffff920a509c00c8) 00000080:00200000:1.0:1788572250.283945:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x175:0x0](ffff920a509bc2c8) 00000080:00200000:1.0:1788572250.283955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x175:0x0](ffff920a509baa08) 00000080:00200000:1.0:1788572250.283964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x176:0x0](ffff920a509adb88) 00000080:00200000:1.0:1788572250.283972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x176:0x0](ffff920a5098c2c8) 00000080:00200000:1.0:1788572250.283981:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x177:0x0](ffff920a509c63c8) 00000080:00200000:1.0:1788572250.283990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x177:0x0](ffff920a509c4b08) 00000080:00200000:1.0:1788572250.283998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x178:0x0](ffff920a509c3248) 00000080:00200000:1.0:1788572250.284005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x178:0x0](ffff920a509c1988) 00000080:00200000:1.0:1788572250.284013:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x179:0x0](ffff920a509bec08) 00000080:00200000:1.0:1788572250.284023:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x179:0x0](ffff920a509bd348) 00000080:00200000:1.0:1788572250.284031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17a:0x0](ffff920a509c21c8) 00000080:00200000:1.0:1788572250.284040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17a:0x0](ffff920a509c1148) 00000080:00200000:1.0:1788572250.284047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17b:0x0](ffff920a509b7448) 00000080:00200000:1.0:1788572250.284055:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17b:0x0](ffff920a509c8908) 00000080:00200000:1.0:1788572250.284063:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17c:0x0](ffff920a509c0908) 00000080:00200000:1.0:1788572250.284071:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17c:0x0](ffff920a509c3a88) 00000080:00200000:1.0:1788572250.284078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17d:0x0](ffff920a509b80c8) 00000080:00200000:1.0:1788572250.284086:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17d:0x0](ffff920a509be3c8) 00000080:00200000:1.0:1788572250.284094:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17e:0x0](ffff920a509b9988) 00000080:00200000:1.0:1788572250.284101:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17e:0x0](ffff920a509ba1c8) 00000080:00200000:1.0:1788572250.284109:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x17f:0x0](ffff920a509c5b88) 00000080:00200000:1.0:1788572250.284191:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x17f:0x0](ffff920a509a42c8) 00000080:00200000:1.0:1788572250.284204:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x180:0x0](ffff920a509bcb08) 00000080:00200000:1.0:1788572250.284212:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x180:0x0](ffff920a509b9148) 00000080:00200000:1.0:1788572250.284219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x181:0x0](ffff920a509caa08) 00000080:00200000:1.0:1788572250.284228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x181:0x0](ffff920a509cdb88) 00000080:00200000:1.0:1788572250.284236:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x182:0x0](ffff920a509dc2c8) 00000080:00200000:1.0:1788572250.284244:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x182:0x0](ffff920a509daa08) 00000080:00200000:1.0:1788572250.284251:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x183:0x0](ffff920a509bf448) 00000080:00200000:1.0:1788572250.284258:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x183:0x0](ffff920a509bb248) 00000080:00200000:1.0:1788572250.284269:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x184:0x0](ffff920a509bdb88) 00000080:00200000:1.0:1788572250.284277:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x184:0x0](ffff920a509b8908) 00000080:00200000:1.0:1788572250.284285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x185:0x0](ffff920a509e5348) 00000080:00200000:1.0:1788572250.284294:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x185:0x0](ffff920a509e3248) 00000080:00200000:1.0:1788572250.284301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x186:0x0](ffff920a509e7448) 00000080:00200000:1.0:1788572250.284309:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x186:0x0](ffff920a509e00c8) 00000080:00200000:1.0:1788572250.284316:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x187:0x0](ffff920a509dd348) 00000080:00200000:1.0:1788572250.284324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x187:0x0](ffff920a509db248) 00000080:00200000:1.0:1788572250.284331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x188:0x0](ffff920a509e63c8) 00000080:00200000:1.0:1788572250.284340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x188:0x0](ffff920a509e4b08) 00000080:00200000:1.0:1788572250.284349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x189:0x0](ffff920a509cd348) 00000080:00200000:1.0:1788572250.284358:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x189:0x0](ffff920a509cb248) 00000080:00200000:1.0:1788572250.284366:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18a:0x0](ffff920a509d80c8) 00000080:00200000:1.0:1788572250.284374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18a:0x0](ffff920a509de3c8) 00000080:00200000:1.0:1788572250.284381:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18b:0x0](ffff920a509cec08) 00000080:00200000:1.0:1788572250.284388:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18b:0x0](ffff920a509cf448) 00000080:00200000:1.0:1788572250.284397:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18c:0x0](ffff920a509e1988) 00000080:00200000:1.0:1788572250.284406:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18c:0x0](ffff920a509e21c8) 00000080:00200000:1.0:1788572250.284414:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18d:0x0](ffff920a509e3a88) 00000080:00200000:1.0:1788572250.284422:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18d:0x0](ffff920a509e0908) 00000080:00200000:1.0:1788572250.284432:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18e:0x0](ffff920a509e1148) 00000080:00200000:1.0:1788572250.284439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18e:0x0](ffff920a509e42c8) 00000080:00200000:1.0:1788572250.284446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x18f:0x0](ffff920a509da1c8) 00000080:00200000:1.0:1788572250.284452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x18f:0x0](ffff920a509d9988) 00000080:00200000:1.0:1788572250.285219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x190:0x0](ffff920a509e2a08) 00000080:00200000:1.0:1788572250.285242:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x190:0x0](ffff920a509e5b88) 00000080:00200000:1.0:1788572250.285251:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x191:0x0](ffff920a509e6c08) 00000080:00200000:1.0:1788572250.285262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x191:0x0](ffff920a509ec2c8) 00000080:00200000:1.0:1788572250.285270:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x192:0x0](ffff920a509ce3c8) 00000080:00200000:1.0:1788572250.285282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x192:0x0](ffff920a509ccb08) 00000080:00200000:1.0:1788572250.285290:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x193:0x0](ffff920a509df448) 00000080:00200000:1.0:1788572250.285298:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x193:0x0](ffff920a509dec08) 00000080:00200000:1.0:1788572250.287139:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x194:0x0](ffff920a509eaa08) 00000080:00200000:1.0:1788572250.287174:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x194:0x0](ffff920a509edb88) 00000080:00200000:1.0:1788572250.287185:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x195:0x0](ffff920a509d1988) 00000080:00200000:1.0:1788572250.287193:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x195:0x0](ffff920a509d21c8) 00000080:00200000:1.0:1788572250.287202:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x196:0x0](ffff920a509ed348) 00000080:00200000:1.0:1788572250.287210:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x196:0x0](ffff920a509eb248) 00000080:00200000:1.0:1788572250.287219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x197:0x0](ffff920a509e80c8) 00000080:00200000:1.0:1788572250.287227:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x197:0x0](ffff920a509ee3c8) 00000080:00200000:1.0:1788572250.287235:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x198:0x0](ffff920a509dba88) 00000080:00200000:1.0:1788572250.287245:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x198:0x0](ffff920a509ddb88) 00000080:00200000:1.0:1788572250.287255:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x199:0x0](ffff920a509c80c8) 00000080:00200000:1.0:1788572250.287268:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x199:0x0](ffff920a509c9988) 00000080:00200000:1.0:1788572250.287277:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19a:0x0](ffff920a509ecb08) 00000080:00200000:1.0:1788572250.287286:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19a:0x0](ffff920a509e9988) 00000080:00200000:1.0:1788572250.287295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19b:0x0](ffff920a509dcb08) 00000080:00200000:1.0:1788572250.287303:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19b:0x0](ffff920a509d8908) 00000080:00200000:1.0:1788572250.287311:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19c:0x0](ffff920a509d9148) 00000080:00200000:1.0:1788572250.287320:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19c:0x0](ffff920a509f21c8) 00000080:00200000:1.0:1788572250.287329:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19d:0x0](ffff920a509ca1c8) 00000080:00200000:1.0:1788572250.287338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19d:0x0](ffff920a509c9148) 00000080:00200000:1.0:1788572250.287346:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19e:0x0](ffff920a509f3a88) 00000080:00200000:1.0:1788572250.287353:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19e:0x0](ffff920a509f0908) 00000080:00200000:1.0:1788572250.287361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x19f:0x0](ffff920a509f42c8) 00000080:00200000:1.0:1788572250.287369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x19f:0x0](ffff920a509f2a08) 00000080:00200000:1.0:1788572250.287378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a0:0x0](ffff920a509f6c08) 00000080:00200000:1.0:1788572250.287387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a0:0x0](ffff920a509f5348) 00000080:00200000:1.0:1788572250.287395:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a1:0x0](ffff920a509e8908) 00000080:00200000:1.0:1788572250.287404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a1:0x0](ffff920a509e9148) 00000080:00200000:1.0:1788572250.287411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a2:0x0](ffff920a509cba88) 00000080:00200000:1.0:1788572250.287420:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a2:0x0](ffff920a509cc2c8) 00000080:00200000:1.0:1788572250.287429:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a3:0x0](ffff920a509f8908) 00000080:00200000:1.0:1788572250.287436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a3:0x0](ffff920a509fc2c8) 00000080:00200000:1.0:1788572250.287444:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a4:0x0](ffff920a509f7448) 00000080:00200000:1.0:1788572250.287453:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a4:0x0](ffff920a509f00c8) 00000080:00200000:1.0:1788572250.287473:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a5:0x0](ffff920a509d3a88) 00000080:00200000:1.0:1788572250.287488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a5:0x0](ffff920a509d0908) 00000080:00200000:1.0:1788572250.287497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a6:0x0](ffff920a509fba88) 00000080:00200000:1.0:1788572250.287506:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a6:0x0](ffff920a509faa08) 00000080:00200000:1.0:1788572250.287514:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a7:0x0](ffff920a509f1988) 00000080:00200000:1.0:1788572250.287523:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a7:0x0](ffff920a509f4b08) 00000080:00200000:1.0:1788572250.287531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a8:0x0](ffff920a509d42c8) 00000080:00200000:1.0:1788572250.287538:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a8:0x0](ffff920a509d2a08) 00000080:00200000:1.0:1788572250.287552:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1a9:0x0](ffff920a509d5b88) 00000080:00200000:1.0:1788572250.287566:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1a9:0x0](ffff920a509d6c08) 00000080:00200000:1.0:1788572250.287574:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1aa:0x0](ffff920a509eec08) 00000080:00200000:1.0:1788572250.287582:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1aa:0x0](ffff920a509ea1c8) 00000080:00200000:1.0:1788572250.287604:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ab:0x0](ffff920a509f3248) 00000080:00200000:1.0:1788572250.287613:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ab:0x0](ffff920a509f5b88) 00000080:00200000:1.0:1788572250.287621:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ac:0x0](ffff920a509f1148) 00000080:00200000:1.0:1788572250.287629:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ac:0x0](ffff920a509f63c8) 00000080:00200000:1.0:1788572250.287637:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ad:0x0](ffff920a509d5348) 00000080:00200000:1.0:1788572250.287644:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ad:0x0](ffff920a509d3248) 00000080:00200000:1.0:1788572250.287653:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ae:0x0](ffff920a509fec08) 00000080:00200000:1.0:1788572250.287662:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ae:0x0](ffff920a509fd348) 00000080:00200000:1.0:1788572250.287670:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1af:0x0](ffff920a509fb248) 00000080:00200000:1.0:1788572250.287677:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1af:0x0](ffff920a509ff448) 00000080:00200000:1.0:1788572250.287686:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b0:0x0](ffff920a50a03248) 00000080:00200000:1.0:1788572250.287696:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b0:0x0](ffff920a50a07448) 00000080:00200000:1.0:1788572250.287704:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b1:0x0](ffff920a509eba88) 00000080:00200000:1.0:1788572250.287712:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b1:0x0](ffff920a509ef448) 00000080:00200000:1.0:1788572250.287720:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b2:0x0](ffff920a50a0c2c8) 00000080:00200000:1.0:1788572250.287728:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b2:0x0](ffff920a50a0aa08) 00000080:00200000:1.0:1788572250.287737:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b3:0x0](ffff920a50a0ec08) 00000080:00200000:1.0:1788572250.287745:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b3:0x0](ffff920a50a0d348) 00000080:00200000:1.0:1788572250.287755:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b4:0x0](ffff920a50a063c8) 00000080:00200000:1.0:1788572250.287764:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b4:0x0](ffff920a50a04b08) 00000080:00200000:1.0:1788572250.287773:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b5:0x0](ffff920a50a01988) 00000080:00200000:1.0:1788572250.287781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b5:0x0](ffff920a50a021c8) 00000080:00200000:1.0:1788572250.287790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b6:0x0](ffff920a50a0b248) 00000080:00200000:1.0:1788572250.287799:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b6:0x0](ffff920a50a0f448) 00000080:00200000:1.0:1788572250.287807:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b7:0x0](ffff920a50a0e3c8) 00000080:00200000:1.0:1788572250.287816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b7:0x0](ffff920a50a0cb08) 00000080:00200000:1.0:1788572250.287823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b8:0x0](ffff920a509d4b08) 00000080:00200000:1.0:1788572250.287830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b8:0x0](ffff920a509d63c8) 00000080:00200000:1.0:1788572250.287838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1b9:0x0](ffff920a50a080c8) 00000080:00200000:1.0:1788572250.287846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1b9:0x0](ffff920a50a09988) 00000080:00200000:1.0:1788572250.287854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ba:0x0](ffff920a509d7448) 00000080:00200000:1.0:1788572250.287862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ba:0x0](ffff920a509d00c8) 00000080:00200000:1.0:1788572250.287869:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1bb:0x0](ffff920a50a03a88) 00000080:00200000:1.0:1788572250.287877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1bb:0x0](ffff920a50a00908) 00000080:00200000:1.0:1788572250.287886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1bc:0x0](ffff920a50a100c8) 00000080:00200000:1.0:1788572250.287894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1bc:0x0](ffff920a50a163c8) 00000080:00200000:1.0:1788572250.287909:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1bd:0x0](ffff920a509f9988) 00000080:00200000:1.0:1788572250.287917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1bd:0x0](ffff920a509fa1c8) 00000080:00200000:1.0:1788572250.287925:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1be:0x0](ffff920a50a042c8) 00000080:00200000:1.0:1788572250.287949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1be:0x0](ffff920a50a01148) 00000080:00200000:1.0:1788572250.287959:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1bf:0x0](ffff920a50a14b08) 00000080:00200000:1.0:1788572250.287967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1bf:0x0](ffff920a50a11988) 00000080:00200000:1.0:1788572250.287976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c0:0x0](ffff920a50a121c8) 00000080:00200000:1.0:1788572250.287984:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c0:0x0](ffff920a50a11148) 00000080:00200000:1.0:1788572250.287997:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c1:0x0](ffff920a50a10908) 00000080:00200000:1.0:1788572250.288006:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c1:0x0](ffff920a50a142c8) 00000080:00200000:1.0:1788572250.288014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c2:0x0](ffff920a50a000c8) 00000080:00200000:1.0:1788572250.288024:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c2:0x0](ffff920a50a05348) 00000080:00200000:1.0:1788572250.288034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c3:0x0](ffff920a50a08908) 00000080:00200000:1.0:1788572250.288043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c3:0x0](ffff920a50a09148) 00000080:00200000:1.0:1788572250.288052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c4:0x0](ffff920a50a0a1c8) 00000080:00200000:1.0:1788572250.288061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c4:0x0](ffff920a50a0db88) 00000080:00200000:1.0:1788572250.288071:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c5:0x0](ffff920a50a06c08) 00000080:00200000:1.0:1788572250.288080:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c5:0x0](ffff920a50a02a08) 00000080:00200000:1.0:1788572250.288088:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c6:0x0](ffff920a50a05b88) 00000080:00200000:1.0:1788572250.288102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c6:0x0](ffff920a50a1c2c8) 00000080:00200000:1.0:1788572250.288111:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c7:0x0](ffff920a50a1aa08) 00000080:00200000:1.0:1788572250.288121:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c7:0x0](ffff920a50a1db88) 00000080:00200000:1.0:1788572250.288130:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c8:0x0](ffff920a50a12a08) 00000080:00200000:1.0:1788572250.288139:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c8:0x0](ffff920a50a15b88) 00000080:00200000:1.0:1788572250.288150:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1c9:0x0](ffff920a50a1d348) 00000080:00200000:1.0:1788572250.288159:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1c9:0x0](ffff920a50a1b248) 00000080:00200000:1.0:1788572250.288169:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ca:0x0](ffff920a50a1f448) 00000080:00200000:1.0:1788572250.288178:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ca:0x0](ffff920a50a180c8) 00000080:00200000:1.0:1788572250.288186:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1cb:0x0](ffff920a50a16c08) 00000080:00200000:1.0:1788572250.288196:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1cb:0x0](ffff920a50a15348) 00000080:00200000:1.0:1788572250.288204:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1cc:0x0](ffff920a50a1cb08) 00000080:00200000:1.0:1788572250.288212:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1cc:0x0](ffff920a50a19988) 00000080:00200000:1.0:1788572250.288221:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1cd:0x0](ffff920a50a0ba88) 00000080:00200000:1.0:1788572250.288230:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1cd:0x0](ffff920a509d1148) 00000080:00200000:1.0:1788572250.288238:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ce:0x0](ffff920a50a221c8) 00000080:00200000:1.0:1788572250.288248:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ce:0x0](ffff920a50a21148) 00000080:00200000:1.0:1788572250.288257:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1cf:0x0](ffff920a50a1ba88) 00000080:00200000:1.0:1788572250.288265:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1cf:0x0](ffff920a50a18908) 00000080:00200000:1.0:1788572250.288273:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d0:0x0](ffff920a50a20908) 00000080:00200000:1.0:1788572250.288282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d0:0x0](ffff920a50a242c8) 00000080:00200000:1.0:1788572250.288291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d1:0x0](ffff920a50a23a88) 00000080:00200000:1.0:1788572250.288301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d1:0x0](ffff920a50a22a08) 00000080:00200000:1.0:1788572250.288311:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d2:0x0](ffff920a50a25b88) 00000080:00200000:1.0:1788572250.288320:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d2:0x0](ffff920a50a26c08) 00000080:00200000:1.0:1788572250.288330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d3:0x0](ffff920a50a1a1c8) 00000080:00200000:1.0:1788572250.288339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d3:0x0](ffff920a50a1e3c8) 00000080:00200000:1.0:1788572250.288348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d4:0x0](ffff920a50a23248) 00000080:00200000:1.0:1788572250.288357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d4:0x0](ffff920a50a27448) 00000080:00200000:1.0:1788572250.288366:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d5:0x0](ffff920a509f80c8) 00000080:00200000:1.0:1788572250.288373:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d5:0x0](ffff920a509fdb88) 00000080:00200000:1.0:1788572250.288381:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d6:0x0](ffff920a50a13248) 00000080:00200000:1.0:1788572250.288391:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d6:0x0](ffff920a50a17448) 00000080:00200000:1.0:1788572250.288400:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d7:0x0](ffff920a50a13a88) 00000080:00200000:1.0:1788572250.288410:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d7:0x0](ffff920a50a29988) 00000080:00200000:1.0:1788572250.288419:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d8:0x0](ffff920a50a263c8) 00000080:00200000:1.0:1788572250.288428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d8:0x0](ffff920a50a24b08) 00000080:00200000:1.0:1788572250.288438:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1d9:0x0](ffff920a50a19148) 00000080:00200000:1.0:1788572250.288447:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1d9:0x0](ffff920a50a1ec08) 00000080:00200000:1.0:1788572250.288456:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1da:0x0](ffff920a50a29148) 00000080:00200000:1.0:1788572250.288466:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1da:0x0](ffff920a50a2ba88) 00000080:00200000:1.0:1788572250.288475:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1db:0x0](ffff920a50a37448) 00000080:00200000:1.0:1788572250.288483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1db:0x0](ffff920a50a300c8) 00000080:00200000:1.0:1788572250.288492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1dc:0x0](ffff920a50a21988) 00000080:00200000:1.0:1788572250.288502:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1dc:0x0](ffff920a50a200c8) 00000080:00200000:1.0:1788572250.288511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1dd:0x0](ffff920a50a25348) 00000080:00200000:1.0:1788572250.288525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1dd:0x0](ffff920a50a3aa08) 00000080:00200000:1.0:1788572250.288534:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1de:0x0](ffff920a50a34b08) 00000080:00200000:1.0:1788572250.288543:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1de:0x0](ffff920a50a31988) 00000080:00200000:1.0:1788572250.288551:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1df:0x0](ffff920a50a3ec08) 00000080:00200000:1.0:1788572250.288560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1df:0x0](ffff920a50a3d348) 00000080:00200000:1.0:1788572250.288570:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e0:0x0](ffff920a509fe3c8) 00000080:00200000:1.0:1788572250.288578:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e0:0x0](ffff920a509fcb08) 00000080:00200000:1.0:1788572250.288606:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e1:0x0](ffff920a50a2c2c8) 00000080:00200000:1.0:1788572250.288618:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e1:0x0](ffff920a50a2aa08) 00000080:00200000:1.0:1788572250.288627:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e2:0x0](ffff920a50a30908) 00000080:00200000:1.0:1788572250.288636:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e2:0x0](ffff920a50a342c8) 00000080:00200000:1.0:1788572250.288645:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e3:0x0](ffff920a50a2d348) 00000080:00200000:1.0:1788572250.288654:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e3:0x0](ffff920a50a2b248) 00000080:00200000:1.0:1788572250.288663:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e4:0x0](ffff920a50a32a08) 00000080:00200000:1.0:1788572250.288671:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e4:0x0](ffff920a50a35b88) 00000080:00200000:1.0:1788572250.288680:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e5:0x0](ffff920a50a3f448) 00000080:00200000:1.0:1788572250.288690:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e5:0x0](ffff920a50a380c8) 00000080:00200000:1.0:1788572250.288699:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e6:0x0](ffff920a50a280c8) 00000080:00200000:1.0:1788572250.288707:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e6:0x0](ffff920a50a2e3c8) 00000080:00200000:1.0:1788572250.288716:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e7:0x0](ffff920a50a3e3c8) 00000080:00200000:1.0:1788572250.288724:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e7:0x0](ffff920a50a3cb08) 00000080:00200000:1.0:1788572250.288733:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e8:0x0](ffff920a50a2f448) 00000080:00200000:1.0:1788572250.288742:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e8:0x0](ffff920a50a2ec08) 00000080:00200000:1.0:1788572250.288751:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1e9:0x0](ffff920a50a3a1c8) 00000080:00200000:1.0:1788572250.288759:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1e9:0x0](ffff920a50a39148) 00000080:00200000:1.0:1788572250.288768:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ea:0x0](ffff920a509f9148) 00000080:00200000:1.0:1788572250.288776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ea:0x0](ffff920a50a421c8) 00000080:00200000:1.0:1788572250.288785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1eb:0x0](ffff920a50a39988) 00000080:00200000:1.0:1788572250.288793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1eb:0x0](ffff920a50a3ba88) 00000080:00200000:1.0:1788572250.288802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ec:0x0](ffff920a50a36c08) 00000080:00200000:1.0:1788572250.288811:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ec:0x0](ffff920a50a33a88) 00000080:00200000:1.0:1788572250.288819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ed:0x0](ffff920a50a38908) 00000080:00200000:1.0:1788572250.288828:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ed:0x0](ffff920a50a3c2c8) 00000080:00200000:1.0:1788572250.288837:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ee:0x0](ffff920a50a3b248) 00000080:00200000:1.0:1788572250.288846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ee:0x0](ffff920a50a3db88) 00000080:00200000:1.0:1788572250.288854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ef:0x0](ffff920a50a41148) 00000080:00200000:1.0:1788572250.288862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ef:0x0](ffff920a50a43a88) 00000080:00200000:1.0:1788572250.288871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f0:0x0](ffff920a50a363c8) 00000080:00200000:1.0:1788572250.288881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f0:0x0](ffff920a50a31148) 00000080:00200000:1.0:1788572250.288890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f1:0x0](ffff920a50a35348) 00000080:00200000:1.0:1788572250.288898:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f1:0x0](ffff920a50a33248) 00000080:00200000:1.0:1788572250.288908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f2:0x0](ffff920a50a321c8) 00000080:00200000:1.0:1788572250.288917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f2:0x0](ffff920a50a521c8) 00000080:00200000:1.0:1788572250.288926:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f3:0x0](ffff920a50a53a88) 00000080:00200000:1.0:1788572250.289855:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f3:0x0](ffff920a50a50908) 00000080:00200000:1.0:1788572250.289865:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f4:0x0](ffff920a50a442c8) 00000080:00200000:1.0:1788572250.289874:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f4:0x0](ffff920a50a42a08) 00000080:00200000:1.0:1788572250.289884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f5:0x0](ffff920a50a52a08) 00000080:00200000:1.0:1788572250.289894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f5:0x0](ffff920a50a55b88) 00000080:00200000:1.0:1788572250.289903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f6:0x0](ffff920a50a4ba88) 00000080:00200000:1.0:1788572250.289914:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f6:0x0](ffff920a50a48908) 00000080:00200000:1.0:1788572250.289926:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f7:0x0](ffff920a50a49148) 00000080:00200000:1.0:1788572250.292344:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f7:0x0](ffff920a50a4c2c8) 00000080:00200000:1.0:1788572250.292361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f8:0x0](ffff920a50a4db88) 00000080:00200000:1.0:1788572250.292372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f8:0x0](ffff920a50a4ec08) 00000080:00200000:1.0:1788572250.292382:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1f9:0x0](ffff920a50a4aa08) 00000080:00200000:1.0:1788572250.292393:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1f9:0x0](ffff920a50a4d348) 00000080:00200000:1.0:1788572250.292404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1fa:0x0](ffff920a50a55348) 00000080:00200000:1.0:1788572250.292418:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1fa:0x0](ffff920a50a53248) 00000080:00200000:1.0:1788572250.292428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1fb:0x0](ffff920a50a2cb08) 00000080:00200000:1.0:1788572250.292444:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1fb:0x0](ffff920a50a2db88) 00000080:00200000:1.0:1788572250.292454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1fc:0x0](ffff920a50a4e3c8) 00000080:00200000:1.0:1788572250.292705:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1fc:0x0](ffff920a50a4cb08) 00000080:00200000:1.0:1788572250.292719:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1fd:0x0](ffff920a50a500c8) 00000080:00200000:1.0:1788572250.292733:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1fd:0x0](ffff920a50a563c8) 00000080:00200000:1.0:1788572250.292743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1fe:0x0](ffff920a50a57448) 00000080:00200000:1.0:1788572250.292752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1fe:0x0](ffff920a50a54b08) 00000080:00200000:1.0:1788572250.292761:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x1ff:0x0](ffff920a50a28908) 00000080:00200000:1.0:1788572250.292771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x1ff:0x0](ffff920a50a2a1c8) 00000080:00200000:1.0:1788572250.292782:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x200:0x0](ffff920a50a480c8) 00000080:00200000:1.0:1788572250.292791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x200:0x0](ffff920a50a4f448) 00000080:00200000:1.0:1788572250.292801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x201:0x0](ffff920a50a45b88) 00000080:00200000:1.0:1788572250.292813:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x201:0x0](ffff920a50a46c08) 00000080:00200000:1.0:1788572250.292822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x202:0x0](ffff920a50a4b248) 00000080:00200000:1.0:1788572250.292834:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x202:0x0](ffff920a50a4a1c8) 00000080:00200000:1.0:1788572250.292844:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x203:0x0](ffff920a50a56c08) 00000080:00200000:1.0:1788572250.292853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x203:0x0](ffff920a50a542c8) 00000080:00200000:1.0:1788572250.292864:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x204:0x0](ffff920a50a5db88) 00000080:00200000:1.0:1788572250.292874:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x204:0x0](ffff920a50a5ec08) 00000080:00200000:1.0:1788572250.292884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x205:0x0](ffff920a50a51988) 00000080:00200000:1.0:1788572250.292894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x205:0x0](ffff920a50a621c8) 00000080:00200000:1.0:1788572250.292904:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x206:0x0](ffff920a50a5b248) 00000080:00200000:1.0:1788572250.292913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x206:0x0](ffff920a50a5f448) 00000080:00200000:1.0:1788572250.292924:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x207:0x0](ffff920a50a63a88) 00000080:00200000:1.0:1788572250.292947:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x207:0x0](ffff920a50a60908) 00000080:00200000:1.0:1788572250.292957:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x208:0x0](ffff920a50a49988) 00000080:00200000:1.0:1788572250.292966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x208:0x0](ffff920a50a6f448) 00000080:00200000:1.0:1788572250.292976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x209:0x0](ffff920a50a680c8) 00000080:00200000:1.0:1788572250.292985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x209:0x0](ffff920a50a6e3c8) 00000080:00200000:1.0:1788572250.292994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20a:0x0](ffff920a50a62a08) 00000080:00200000:1.0:1788572250.293004:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20a:0x0](ffff920a50a65b88) 00000080:00200000:1.0:1788572250.293014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20b:0x0](ffff920a50a65348) 00000080:00200000:1.0:1788572250.293024:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20b:0x0](ffff920a50a63248) 00000080:00200000:1.0:1788572250.293034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20c:0x0](ffff920a50a66c08) 00000080:00200000:1.0:1788572250.293043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20c:0x0](ffff920a50a67448) 00000080:00200000:1.0:1788572250.293053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20d:0x0](ffff920a50a6cb08) 00000080:00200000:1.0:1788572250.293063:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20d:0x0](ffff920a50a69988) 00000080:00200000:1.0:1788572250.293073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20e:0x0](ffff920a50a64b08) 00000080:00200000:1.0:1788572250.293081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20e:0x0](ffff920a50a61988) 00000080:00200000:1.0:1788572250.293091:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x20f:0x0](ffff920a50a663c8) 00000080:00200000:1.0:1788572250.293101:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x20f:0x0](ffff920a50a600c8) 00000080:00200000:1.0:1788572250.293111:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x210:0x0](ffff920a50a69148) 00000080:00200000:1.0:1788572250.294061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x210:0x0](ffff920a50a6ba88) 00000080:00200000:1.0:1788572250.294079:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x211:0x0](ffff920a50a61148) 00000080:00200000:1.0:1788572250.294089:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x211:0x0](ffff920a50a642c8) 00000080:00200000:1.0:1788572250.294102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x212:0x0](ffff920a50a45348) 00000080:00200000:1.0:1788572250.294112:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x212:0x0](ffff920a50a43248) 00000080:00200000:1.0:1788572250.294197:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x213:0x0](ffff920a50a76c08) 00000080:00200000:1.0:1788572250.294208:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x213:0x0](ffff920a50a75348) 00000080:00200000:1.0:1788572250.294218:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x214:0x0](ffff920a50a75b88) 00000080:00200000:1.0:1788572250.294226:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x214:0x0](ffff920a50a73248) 00000080:00200000:1.0:1788572250.294235:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x215:0x0](ffff920a50a77448) 00000080:00200000:1.0:1788572250.294245:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x215:0x0](ffff920a50a700c8) 00000080:00200000:1.0:1788572250.294254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x216:0x0](ffff920a50a6c2c8) 00000080:00200000:1.0:1788572250.294265:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x216:0x0](ffff920a50a6aa08) 00000080:00200000:1.0:1788572250.294274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x217:0x0](ffff920a50a71988) 00000080:00200000:1.0:1788572250.294284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x217:0x0](ffff920a50a721c8) 00000080:00200000:1.0:1788572250.294292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x218:0x0](ffff920a50a74b08) 00000080:00200000:1.0:1788572250.294301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x218:0x0](ffff920a50a71148) 00000080:00200000:1.0:1788572250.294311:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x219:0x0](ffff920a50a6ec08) 00000080:00200000:1.0:1788572250.294322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x219:0x0](ffff920a50a6d348) 00000080:00200000:1.0:1788572250.294331:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21a:0x0](ffff920a50a6b248) 00000080:00200000:1.0:1788572250.294340:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21a:0x0](ffff920a50a6db88) 00000080:00200000:1.0:1788572250.294349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21b:0x0](ffff920a50a742c8) 00000080:00200000:1.0:1788572250.294358:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21b:0x0](ffff920a50a70908) 00000080:00200000:1.0:1788572250.294366:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21c:0x0](ffff920a50a73a88) 00000080:00200000:1.0:1788572250.294374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21c:0x0](ffff920a50a763c8) 00000080:00200000:1.0:1788572250.294383:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21d:0x0](ffff920a50a68908) 00000080:00200000:1.0:1788572250.294392:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21d:0x0](ffff920a50a6a1c8) 00000080:00200000:1.0:1788572250.294402:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21e:0x0](ffff920a50a463c8) 00000080:00200000:1.0:1788572250.294412:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21e:0x0](ffff920a50a44b08) 00000080:00200000:1.0:1788572250.294421:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x21f:0x0](ffff920a50a5e3c8) 00000080:00200000:1.0:1788572250.294433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x21f:0x0](ffff920a50a5cb08) 00000080:00200000:1.0:1788572250.294443:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x220:0x0](ffff920a50a72a08) 00000080:00200000:1.0:1788572250.294452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x220:0x0](ffff920a50a80908) 00000080:00200000:1.0:1788572250.294682:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x221:0x0](ffff920a50a7aa08) 00000080:00200000:1.0:1788572250.294708:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x221:0x0](ffff920a50a7db88) 00000080:00200000:1.0:1788572250.294718:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x222:0x0](ffff920a50a41988) 00000080:00200000:1.0:1788572250.294729:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x222:0x0](ffff920a50a400c8) 00000080:00200000:1.0:1788572250.294738:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x223:0x0](ffff920a50a5a1c8) 00000080:00200000:1.0:1788572250.294749:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x223:0x0](ffff920a50a59148) 00000080:00200000:1.0:1788572250.294758:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x224:0x0](ffff920a50a5ba88) 00000080:00200000:1.0:1788572250.294768:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x224:0x0](ffff920a50a58908) 00000080:00200000:1.0:1788572250.294776:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x225:0x0](ffff920a50a85b88) 00000080:00200000:1.0:1788572250.294786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x225:0x0](ffff920a50a86c08) 00000080:00200000:1.0:1788572250.294796:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x226:0x0](ffff920a50a7f448) 00000080:00200000:1.0:1788572250.294806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x226:0x0](ffff920a50a780c8) 00000080:00200000:1.0:1788572250.294816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x227:0x0](ffff920a50a82a08) 00000080:00200000:1.0:1788572250.294825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x227:0x0](ffff920a50a85348) 00000080:00200000:1.0:1788572250.294833:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x228:0x0](ffff920a50a87448) 00000080:00200000:1.0:1788572250.294842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x228:0x0](ffff920a50a800c8) 00000080:00200000:1.0:1788572250.294852:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x229:0x0](ffff920a50a5c2c8) 00000080:00200000:1.0:1788572250.294862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x229:0x0](ffff920a50a5aa08) 00000080:00200000:1.0:1788572250.294871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22a:0x0](ffff920a50a7e3c8) 00000080:00200000:1.0:1788572250.294880:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22a:0x0](ffff920a50a7cb08) 00000080:00200000:1.0:1788572250.294889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22b:0x0](ffff920a50a7a1c8) 00000080:00200000:1.0:1788572250.294899:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22b:0x0](ffff920a50a79148) 00000080:00200000:1.0:1788572250.294908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22c:0x0](ffff920a50a59988) 00000080:00200000:1.0:1788572250.294916:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22c:0x0](ffff920a50a580c8) 00000080:00200000:1.0:1788572250.294926:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22d:0x0](ffff920a50a79988) 00000080:00200000:1.0:1788572250.294948:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22d:0x0](ffff920a50a7ba88) 00000080:00200000:1.0:1788572250.294958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22e:0x0](ffff920a50a821c8) 00000080:00200000:1.0:1788572250.294967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22e:0x0](ffff920a50a81148) 00000080:00200000:1.0:1788572250.294976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x22f:0x0](ffff920a50a78908) 00000080:00200000:1.0:1788572250.294986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x22f:0x0](ffff920a50a7b248) 00000080:00200000:1.0:1788572250.294995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x230:0x0](ffff920a50a83a88) 00000080:00200000:1.0:1788572250.295005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x230:0x0](ffff920a50a81988) 00000080:00200000:1.0:1788572250.295014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x231:0x0](ffff920a50a7d348) 00000080:00200000:1.0:1788572250.295022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x231:0x0](ffff920a50a7ec08) 00000080:00200000:1.0:1788572250.295031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x232:0x0](ffff920a50a7c2c8) 00000080:00200000:1.0:1788572250.295234:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x232:0x0](ffff920a50a8db88) 00000080:00200000:1.0:1788572250.295248:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x233:0x0](ffff920a50a8d348) 00000080:00200000:1.0:1788572250.295257:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x233:0x0](ffff920a50a8b248) 00000080:00200000:1.0:1788572250.295265:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x234:0x0](ffff920a50a40908) 00000080:00200000:1.0:1788572250.295275:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x234:0x0](ffff920a50a47448) 00000080:00200000:1.0:1788572250.295284:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x235:0x0](ffff920a50a880c8) 00000080:00200000:1.0:1788572250.295294:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x235:0x0](ffff920a50a8e3c8) 00000080:00200000:1.0:1788572250.295302:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x236:0x0](ffff920a50a5d348) 00000080:00200000:1.0:1788572250.295319:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x236:0x0](ffff920a50a91988) 00000080:00200000:1.0:1788572250.295328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x237:0x0](ffff920a50a921c8) 00000080:00200000:1.0:1788572250.295337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x237:0x0](ffff920a50a91148) 00000080:00200000:1.0:1788572250.295346:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x238:0x0](ffff920a50a93a88) 00000080:00200000:1.0:1788572250.295355:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x238:0x0](ffff920a50a90908) 00000080:00200000:1.0:1788572250.295363:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x239:0x0](ffff920a50a9cb08) 00000080:00200000:1.0:1788572250.295372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x239:0x0](ffff920a50a99988) 00000080:00200000:1.0:1788572250.295381:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23a:0x0](ffff920a50a9a1c8) 00000080:00200000:1.0:1788572250.295391:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23a:0x0](ffff920a50a99148) 00000080:00200000:1.0:1788572250.295399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23b:0x0](ffff920a50a9ba88) 00000080:00200000:1.0:1788572250.295407:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23b:0x0](ffff920a50a98908) 00000080:00200000:1.0:1788572250.295416:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23c:0x0](ffff920a50a8a1c8) 00000080:00200000:1.0:1788572250.295425:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23c:0x0](ffff920a50a89148) 00000080:00200000:1.0:1788572250.295434:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23d:0x0](ffff920a50a83248) 00000080:00200000:1.0:1788572250.295443:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23d:0x0](ffff920a50a863c8) 00000080:00200000:1.0:1788572250.295451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23e:0x0](ffff920a50a88908) 00000080:00200000:1.0:1788572250.295751:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23e:0x0](ffff920a50a8c2c8) 00000080:00200000:1.0:1788572250.295767:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x23f:0x0](ffff920a50a8aa08) 00000080:00200000:1.0:1788572250.295778:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x23f:0x0](ffff920a50a8ba88) 00000080:00200000:1.0:1788572250.295787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x240:0x0](ffff920a50a842c8) 00000080:00200000:1.0:1788572250.295797:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x240:0x0](ffff920a50aa63c8) 00000080:00200000:1.0:1788572250.295806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x241:0x0](ffff920a50aa1988) 00000080:00200000:1.0:1788572250.295816:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x241:0x0](ffff920a50aa21c8) 00000080:00200000:1.0:1788572250.295825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x242:0x0](ffff920a50a942c8) 00000080:00200000:1.0:1788572250.295834:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x242:0x0](ffff920a50a92a08) 00000080:00200000:1.0:1788572250.295844:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x243:0x0](ffff920a50a8cb08) 00000080:00200000:1.0:1788572250.295853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x243:0x0](ffff920a50a8f448) 00000080:00200000:1.0:1788572250.295862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x244:0x0](ffff920a50a9c2c8) 00000080:00200000:1.0:1788572250.295872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x244:0x0](ffff920a50a9aa08) 00000080:00200000:1.0:1788572250.295881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x245:0x0](ffff920a50a95b88) 00000080:00200000:1.0:1788572250.295890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x245:0x0](ffff920a50a96c08) 00000080:00200000:1.0:1788572250.295901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x246:0x0](ffff920a50a8ec08) 00000080:00200000:1.0:1788572250.295911:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x246:0x0](ffff920a50aad348) 00000080:00200000:1.0:1788572250.295920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x247:0x0](ffff920a50a9ec08) 00000080:00200000:1.0:1788572250.295940:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x247:0x0](ffff920a50a9d348) 00000080:00200000:1.0:1788572250.295950:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x248:0x0](ffff920a50aab248) 00000080:00200000:1.0:1788572250.295960:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x248:0x0](ffff920a50aaf448) 00000080:00200000:1.0:1788572250.295969:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x249:0x0](ffff920a50aa80c8) 00000080:00200000:1.0:1788572250.295977:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x249:0x0](ffff920a50aae3c8) 00000080:00200000:1.0:1788572250.295986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24a:0x0](ffff920a50aa42c8) 00000080:00200000:1.0:1788572250.295995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24a:0x0](ffff920a50aa2a08) 00000080:00200000:1.0:1788572250.296004:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24b:0x0](ffff920a50aa6c08) 00000080:00200000:1.0:1788572250.296013:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24b:0x0](ffff920a50aa5348) 00000080:00200000:1.0:1788572250.296022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24c:0x0](ffff920a50aaa1c8) 00000080:00200000:1.0:1788572250.296032:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24c:0x0](ffff920a50aa9148) 00000080:00200000:1.0:1788572250.296041:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24d:0x0](ffff920a50aaba88) 00000080:00200000:1.0:1788572250.296049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24d:0x0](ffff920a50aa8908) 00000080:00200000:1.0:1788572250.296058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24e:0x0](ffff920a50aa3248) 00000080:00200000:1.0:1788572250.296069:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24e:0x0](ffff920a50aa7448) 00000080:00200000:1.0:1788572250.296079:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x24f:0x0](ffff920a50aa00c8) 00000080:00200000:1.0:1788572250.296090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x24f:0x0](ffff920a50aa5b88) 00000080:00200000:1.0:1788572250.296100:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x250:0x0](ffff920a50a9b248) 00000080:00200000:1.0:1788572250.296109:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x250:0x0](ffff920a50a9f448) 00000080:00200000:1.0:1788572250.296194:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x251:0x0](ffff920a50aadb88) 00000080:00200000:1.0:1788572250.296207:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x251:0x0](ffff920a50aaec08) 00000080:00200000:1.0:1788572250.296216:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x252:0x0](ffff920a50aa3a88) 00000080:00200000:1.0:1788572250.296225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x252:0x0](ffff920a50aa1148) 00000080:00200000:1.0:1788572250.296233:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x253:0x0](ffff920a50a95348) 00000080:00200000:1.0:1788572250.296243:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x253:0x0](ffff920a50a93248) 00000080:00200000:1.0:1788572250.296252:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x254:0x0](ffff920a50aa0908) 00000080:00200000:1.0:1788572250.296261:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x254:0x0](ffff920a50a84b08) 00000080:00200000:1.0:1788572250.296271:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x255:0x0](ffff920a50aac2c8) 00000080:00200000:1.0:1788572250.296280:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x255:0x0](ffff920a50aa9988) 00000080:00200000:1.0:1788572250.296290:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x256:0x0](ffff920a50aacb08) 00000080:00200000:1.0:1788572250.296300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x256:0x0](ffff920a50aaaa08) 00000080:00200000:1.0:1788572250.296309:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x257:0x0](ffff920a50a9e3c8) 00000080:00200000:1.0:1788572250.296319:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x257:0x0](ffff920a50a9db88) 00000080:00200000:1.0:1788572250.296329:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x258:0x0](ffff920a50ab4b08) 00000080:00200000:1.0:1788572250.296338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x258:0x0](ffff920a50ab1988) 00000080:00200000:1.0:1788572250.296349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x259:0x0](ffff920a50a89988) 00000080:00200000:1.0:1788572250.296358:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x259:0x0](ffff920a50abb248) 00000080:00200000:1.0:1788572250.296367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25a:0x0](ffff920a50ab80c8) 00000080:00200000:1.0:1788572250.296377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25a:0x0](ffff920a50abe3c8) 00000080:00200000:1.0:1788572250.296386:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25b:0x0](ffff920a50a97448) 00000080:00200000:1.0:1788572250.296396:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25b:0x0](ffff920a50a900c8) 00000080:00200000:1.0:1788572250.296404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25c:0x0](ffff920a50ab3a88) 00000080:00200000:1.0:1788572250.296413:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25c:0x0](ffff920a50ab0908) 00000080:00200000:1.0:1788572250.296422:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25d:0x0](ffff920a50abf448) 00000080:00200000:1.0:1788572250.296431:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25d:0x0](ffff920a50abcb08) 00000080:00200000:1.0:1788572250.296440:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25e:0x0](ffff920a50ab2a08) 00000080:00200000:1.0:1788572250.296449:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25e:0x0](ffff920a50ab5b88) 00000080:00200000:1.0:1788572250.296554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x25f:0x0](ffff920a50aba1c8) 00000080:00200000:1.0:1788572250.296573:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x25f:0x0](ffff920a50ab9148) 00000080:00200000:1.0:1788572250.296701:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x260:0x0](ffff920a50ab6c08) 00000080:00200000:1.0:1788572250.296715:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x260:0x0](ffff920a50ab5348) 00000080:00200000:1.0:1788572250.296724:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x261:0x0](ffff920a50ab9988) 00000080:00200000:1.0:1788572250.296733:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x261:0x0](ffff920a50abba88) 00000080:00200000:1.0:1788572250.296743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x262:0x0](ffff920a50ab3248) 00000080:00200000:1.0:1788572250.296751:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x262:0x0](ffff920a50ab42c8) 00000080:00200000:1.0:1788572250.296760:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x263:0x0](ffff920a50a963c8) 00000080:00200000:1.0:1788572250.296769:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x263:0x0](ffff920a50a94b08) 00000080:00200000:1.0:1788572250.296779:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x264:0x0](ffff920a50abaa08) 00000080:00200000:1.0:1788572250.296789:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x264:0x0](ffff920a50abdb88) 00000080:00200000:1.0:1788572250.296799:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x265:0x0](ffff920a50ab1148) 00000080:00200000:1.0:1788572250.296809:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x265:0x0](ffff920a50ab21c8) 00000080:00200000:1.0:1788572250.296818:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x266:0x0](ffff920a50ab63c8) 00000080:00200000:1.0:1788572250.296827:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x266:0x0](ffff920a50ab00c8) 00000080:00200000:1.0:1788572250.296836:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x267:0x0](ffff920a50aa4b08) 00000080:00200000:1.0:1788572250.296846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x267:0x0](ffff920a50ac3248) 00000080:00200000:1.0:1788572250.296854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x268:0x0](ffff920a50ac00c8) 00000080:00200000:1.0:1788572250.296863:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x268:0x0](ffff920a50ac63c8) 00000080:00200000:1.0:1788572250.296873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x269:0x0](ffff920a50abec08) 00000080:00200000:1.0:1788572250.296882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x269:0x0](ffff920a50abd348) 00000080:00200000:1.0:1788572250.296891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26a:0x0](ffff920a50ac4b08) 00000080:00200000:1.0:1788572250.296900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26a:0x0](ffff920a50ac1988) 00000080:00200000:1.0:1788572250.296909:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26b:0x0](ffff920a50accb08) 00000080:00200000:1.0:1788572250.296918:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26b:0x0](ffff920a50ac9988) 00000080:00200000:1.0:1788572250.296939:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26c:0x0](ffff920a50abc2c8) 00000080:00200000:1.0:1788572250.296950:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26c:0x0](ffff920a50ab8908) 00000080:00200000:1.0:1788572250.296958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26d:0x0](ffff920a50ad5348) 00000080:00200000:1.0:1788572250.296967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26d:0x0](ffff920a50ad3248) 00000080:00200000:1.0:1788572250.296976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26e:0x0](ffff920a50ad7448) 00000080:00200000:1.0:1788572250.296986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26e:0x0](ffff920a50ad00c8) 00000080:00200000:1.0:1788572250.296995:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x26f:0x0](ffff920a50ac42c8) 00000080:00200000:1.0:1788572250.297007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x26f:0x0](ffff920a50ac2a08) 00000080:00200000:1.0:1788572250.297017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x270:0x0](ffff920a50ac5b88) 00000080:00200000:1.0:1788572250.297025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x270:0x0](ffff920a50ac6c08) 00000080:00200000:1.0:1788572250.297035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x271:0x0](ffff920a50a51148) 00000080:00200000:1.0:1788572250.297047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x271:0x0](ffff920a50adec08) 00000080:00200000:1.0:1788572250.297057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x272:0x0](ffff920a50aca1c8) 00000080:00200000:1.0:1788572250.297066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x272:0x0](ffff920a50ac9148) 00000080:00200000:1.0:1788572250.297077:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x273:0x0](ffff920a50ad1988) 00000080:00200000:1.0:1788572250.297086:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x273:0x0](ffff920a50ad21c8) 00000080:00200000:1.0:1788572250.297096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x274:0x0](ffff920a50ac8908) 00000080:00200000:1.0:1788572250.297105:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x274:0x0](ffff920a50acc2c8) 00000080:00200000:1.0:1788572250.297114:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x275:0x0](ffff920a50acaa08) 00000080:00200000:1.0:1788572250.297209:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x275:0x0](ffff920a50acdb88) 00000080:00200000:1.0:1788572250.297219:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x276:0x0](ffff920a50acec08) 00000080:00200000:1.0:1788572250.297230:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x276:0x0](ffff920a50acd348) 00000080:00200000:1.0:1788572250.297239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x277:0x0](ffff920a50ac3a88) 00000080:00200000:1.0:1788572250.297250:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x277:0x0](ffff920a50ac1148) 00000080:00200000:1.0:1788572250.297259:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x278:0x0](ffff920a50ac7448) 00000080:00200000:1.0:1788572250.297269:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x278:0x0](ffff920a50ac0908) 00000080:00200000:1.0:1788572250.297279:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x279:0x0](ffff920a50ad0908) 00000080:00200000:1.0:1788572250.297289:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x279:0x0](ffff920a50ad42c8) 00000080:00200000:1.0:1788572250.297299:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27a:0x0](ffff920a50ad2a08) 00000080:00200000:1.0:1788572250.297308:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27b:0x0](ffff920a50ad5b88) 00000080:00200000:1.0:1788572250.297317:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27b:0x0](ffff920a50ac5348) 00000080:00200000:1.0:1788572250.297328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27c:0x0](ffff920a50ac21c8) 00000080:00200000:1.0:1788572250.297337:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27c:0x0](ffff920a50ad6c08) 00000080:00200000:1.0:1788572250.297347:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27d:0x0](ffff920a50ad3a88) 00000080:00200000:1.0:1788572250.297356:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27d:0x0](ffff920a50ae63c8) 00000080:00200000:1.0:1788572250.297365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27e:0x0](ffff920a50ae4b08) 00000080:00200000:1.0:1788572250.297374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27e:0x0](ffff920a50adf448) 00000080:00200000:1.0:1788572250.297385:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x27f:0x0](ffff920a50ad80c8) 00000080:00200000:1.0:1788572250.297395:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x27f:0x0](ffff920a50ae21c8) 00000080:00200000:1.0:1788572250.297404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x280:0x0](ffff920a50ae1148) 00000080:00200000:1.0:1788572250.297477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x280:0x0](ffff920a50adb248) 00000080:00200000:1.0:1788572250.297492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x281:0x0](ffff920a50ade3c8) 00000080:00200000:1.0:1788572250.297501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x281:0x0](ffff920a50ae3a88) 00000080:00200000:1.0:1788572250.297511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x282:0x0](ffff920a50ae0908) 00000080:00200000:1.0:1788572250.297521:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x282:0x0](ffff920a50ad4b08) 00000080:00200000:1.0:1788572250.297532:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x283:0x0](ffff920a50ad63c8) 00000080:00200000:1.0:1788572250.297541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x283:0x0](ffff920a50aea1c8) 00000080:00200000:1.0:1788572250.297550:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x284:0x0](ffff920a50ae9148) 00000080:00200000:1.0:1788572250.297560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x284:0x0](ffff920a50acb248) 00000080:00200000:1.0:1788572250.297569:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x285:0x0](ffff920a50acf448) 00000080:00200000:1.0:1788572250.297577:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x285:0x0](ffff920a50acba88) 00000080:00200000:1.0:1788572250.297618:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x286:0x0](ffff920a50ace3c8) 00000080:00200000:1.0:1788572250.297629:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x286:0x0](ffff920a50ac80c8) 00000080:00200000:1.0:1788572250.297638:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x287:0x0](ffff920a50af42c8) 00000080:00200000:1.0:1788572250.297648:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x287:0x0](ffff920a50aeaa08) 00000080:00200000:1.0:1788572250.297658:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x288:0x0](ffff920a50aedb88) 00000080:00200000:1.0:1788572250.297667:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x288:0x0](ffff920a50ae5b88) 00000080:00200000:1.0:1788572250.297676:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x289:0x0](ffff920a50ae6c08) 00000080:00200000:1.0:1788572250.297685:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x289:0x0](ffff920a50ae5348) 00000080:00200000:1.0:1788572250.297695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28a:0x0](ffff920a50ae3248) 00000080:00200000:1.0:1788572250.297704:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28a:0x0](ffff920a50ae7448) 00000080:00200000:1.0:1788572250.297712:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28b:0x0](ffff920a50ae00c8) 00000080:00200000:1.0:1788572250.297721:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28b:0x0](ffff920a50ae2a08) 00000080:00200000:1.0:1788572250.297730:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28c:0x0](ffff920a50ae42c8) 00000080:00200000:1.0:1788572250.297752:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28c:0x0](ffff920a50af2a08) 00000080:00200000:1.0:1788572250.297763:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28d:0x0](ffff920a50af5b88) 00000080:00200000:1.0:1788572250.297772:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28d:0x0](ffff920a50ad1148) 00000080:00200000:1.0:1788572250.297783:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28e:0x0](ffff920a50ab7448) 00000080:00200000:1.0:1788572250.297792:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28e:0x0](ffff920a50ae1988) 00000080:00200000:1.0:1788572250.297802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x28f:0x0](ffff920a50afdb88) 00000080:00200000:1.0:1788572250.297811:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x28f:0x0](ffff920a50aee3c8) 00000080:00200000:1.0:1788572250.297821:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x290:0x0](ffff920a50aecb08) 00000080:00200000:1.0:1788572250.297830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x290:0x0](ffff920a50ae9988) 00000080:00200000:1.0:1788572250.297839:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x291:0x0](ffff920a50ae80c8) 00000080:00200000:1.0:1788572250.297849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x291:0x0](ffff920a50aeb248) 00000080:00200000:1.0:1788572250.297858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x292:0x0](ffff920a50aed348) 00000080:00200000:1.0:1788572250.297867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x292:0x0](ffff920a50aef448) 00000080:00200000:1.0:1788572250.297875:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x293:0x0](ffff920a50aeec08) 00000080:00200000:1.0:1788572250.297884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x293:0x0](ffff920a50af5348) 00000080:00200000:1.0:1788572250.297892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x294:0x0](ffff920a50af3248) 00000080:00200000:1.0:1788572250.297939:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x294:0x0](ffff920a50afb248) 00000080:00200000:1.0:1788572250.297955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x295:0x0](ffff920a50aff448) 00000080:00200000:1.0:1788572250.297964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x295:0x0](ffff920a50ae8908) 00000080:00200000:1.0:1788572250.297973:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x296:0x0](ffff920a50aeba88) 00000080:00200000:1.0:1788572250.297981:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x296:0x0](ffff920a50af80c8) 00000080:00200000:1.0:1788572250.297990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x297:0x0](ffff920a50afe3c8) 00000080:00200000:1.0:1788572250.297998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x297:0x0](ffff920a50ada1c8) 00000080:00200000:1.0:1788572250.298008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x298:0x0](ffff920a50ad9148) 00000080:00200000:1.0:1788572250.298017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x298:0x0](ffff920a50afcb08) 00000080:00200000:1.0:1788572250.298026:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x299:0x0](ffff920a50af9988) 00000080:00200000:1.0:1788572250.298034:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x299:0x0](ffff920a50aec2c8) 00000080:00200000:1.0:1788572250.298042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29a:0x0](ffff920a50b01148) 00000080:00200000:1.0:1788572250.298051:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29a:0x0](ffff920a50af7448) 00000080:00200000:1.0:1788572250.298060:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29b:0x0](ffff920a50af00c8) 00000080:00200000:1.0:1788572250.298070:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29b:0x0](ffff920a50b00908) 00000080:00200000:1.0:1788572250.298078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29c:0x0](ffff920a50b042c8) 00000080:00200000:1.0:1788572250.298087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29c:0x0](ffff920a50afba88) 00000080:00200000:1.0:1788572250.298096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29d:0x0](ffff920a50af8908) 00000080:00200000:1.0:1788572250.298104:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29d:0x0](ffff920a50afc2c8) 00000080:00200000:1.0:1788572250.298113:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29e:0x0](ffff920a50afaa08) 00000080:00200000:1.0:1788572250.298445:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29e:0x0](ffff920a50af9148) 00000080:00200000:1.0:1788572250.298455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x29f:0x0](ffff920a50afa1c8) 00000080:00200000:1.0:1788572250.298470:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x29f:0x0](ffff920a50afd348) 00000080:00200000:1.0:1788572250.298479:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a0:0x0](ffff920a50afec08) 00000080:00200000:1.0:1788572250.298488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a0:0x0](ffff920a50b05b88) 00000080:00200000:1.0:1788572250.298499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a1:0x0](ffff920a50b06c08) 00000080:00200000:1.0:1788572250.298509:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a2:0x0](ffff920a50b0c2c8) 00000080:00200000:1.0:1788572250.298517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a2:0x0](ffff920a50b0aa08) 00000080:00200000:1.0:1788572250.298526:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a3:0x0](ffff920a50b0ec08) 00000080:00200000:1.0:1788572250.298535:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a3:0x0](ffff920a50b0d348) 00000080:00200000:1.0:1788572250.298543:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a4:0x0](ffff920a50b0db88) 00000080:00200000:1.0:1788572250.298553:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a4:0x0](ffff920a50b0b248) 00000080:00200000:1.0:1788572250.298562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a5:0x0](ffff920a50b07448) 00000080:00200000:1.0:1788572250.298571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a5:0x0](ffff920a50b000c8) 00000080:00200000:1.0:1788572250.298581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a6:0x0](ffff920a50b080c8) 00000080:00200000:1.0:1788572250.299552:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a6:0x0](ffff920a50b0e3c8) 00000080:00200000:1.0:1788572250.299567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a7:0x0](ffff920a50af4b08) 00000080:00200000:1.0:1788572250.299579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a7:0x0](ffff920a50af1988) 00000080:00200000:1.0:1788572250.299806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a8:0x0](ffff920a50b063c8) 00000080:00200000:1.0:1788572250.299821:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a8:0x0](ffff920a50b04b08) 00000080:00200000:1.0:1788572250.299830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2a9:0x0](ffff920a50b0cb08) 00000080:00200000:1.0:1788572250.299839:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2a9:0x0](ffff920a50b09988) 00000080:00200000:1.0:1788572250.299849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2aa:0x0](ffff920a50b021c8) 00000080:00200000:1.0:1788572250.299859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2aa:0x0](ffff920a50b03248) 00000080:00200000:1.0:1788572250.299869:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ab:0x0](ffff920a50b01988) 00000080:00200000:1.0:1788572250.299878:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ab:0x0](ffff920a50b05348) 00000080:00200000:1.0:1788572250.299889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ac:0x0](ffff920a50addb88) 00000080:00200000:1.0:1788572250.299903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ac:0x0](ffff920a50adc2c8) 00000080:00200000:1.0:1788572250.299913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ad:0x0](ffff920a50b09148) 00000080:00200000:1.0:1788572250.299923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ad:0x0](ffff920a50b0ba88) 00000080:00200000:1.0:1788572250.299949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ae:0x0](ffff920a50b0a1c8) 00000080:00200000:1.0:1788572250.299984:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ae:0x0](ffff920a50b0f448) 00000080:00200000:1.0:1788572250.299998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2af:0x0](ffff920a50b02a08) 00000080:00200000:1.0:1788572250.300008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2af:0x0](ffff920a50b15348) 00000080:00200000:1.0:1788572250.300017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b0:0x0](ffff920a50af1148) 00000080:00200000:1.0:1788572250.300027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b0:0x0](ffff920a50af3a88) 00000080:00200000:1.0:1788572250.300037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b1:0x0](ffff920a50b13248) 00000080:00200000:1.0:1788572250.300047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b1:0x0](ffff920a50b17448) 00000080:00200000:1.0:1788572250.300057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b2:0x0](ffff920a50adaa08) 00000080:00200000:1.0:1788572250.300067:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b2:0x0](ffff920a50ad8908) 00000080:00200000:1.0:1788572250.300078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b3:0x0](ffff920a50b100c8) 00000080:00200000:1.0:1788572250.300087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b3:0x0](ffff920a50b163c8) 00000080:00200000:1.0:1788572250.300097:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b4:0x0](ffff920a50b18908) 00000080:00200000:1.0:1788572250.300107:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b4:0x0](ffff920a50b1c2c8) 00000080:00200000:1.0:1788572250.300287:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b5:0x0](ffff920a50b11988) 00000080:00200000:1.0:1788572250.300303:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b5:0x0](ffff920a50b121c8) 00000080:00200000:1.0:1788572250.300313:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b6:0x0](ffff920a50af0908) 00000080:00200000:1.0:1788572250.300324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b6:0x0](ffff920a50af21c8) 00000080:00200000:1.0:1788572250.300334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b7:0x0](ffff920a50b14b08) 00000080:00200000:1.0:1788572250.300343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b7:0x0](ffff920a50b11148) 00000080:00200000:1.0:1788572250.300352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b8:0x0](ffff920a50b1ec08) 00000080:00200000:1.0:1788572250.300362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b8:0x0](ffff920a50b1d348) 00000080:00200000:1.0:1788572250.300371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2b9:0x0](ffff920a50b10908) 00000080:00200000:1.0:1788572250.300392:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2b9:0x0](ffff920a50b142c8) 00000080:00200000:1.0:1788572250.300402:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ba:0x0](ffff920a50b180c8) 00000080:00200000:1.0:1788572250.300413:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ba:0x0](ffff920a50b1e3c8) 00000080:00200000:1.0:1788572250.300424:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2bb:0x0](ffff920a50b15b88) 00000080:00200000:1.0:1788572250.300433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2bb:0x0](ffff920a50b16c08) 00000080:00200000:1.0:1788572250.300443:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2bc:0x0](ffff920a50af63c8) 00000080:00200000:1.0:1788572250.300455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2bc:0x0](ffff920a50af6c08) 00000080:00200000:2.0:1788572250.300966:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2bd:0x0](ffff920a50b12a08) 00000080:00200000:2.0:1788572250.300983:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2bd:0x0](ffff920a50b13a88) 00000080:00200000:2.0:1788572250.300993:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2be:0x0](ffff920a50b1a1c8) 00000080:00200000:2.0:1788572250.301003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2be:0x0](ffff920a50b19148) 00000080:00200000:2.0:1788572250.301012:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2bf:0x0](ffff920a50b19988) 00000080:00200000:2.0:1788572250.301022:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2bf:0x0](ffff920a50b1ba88) 00000080:00200000:2.0:1788572250.301032:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c0:0x0](ffff920a50b23248) 00000080:00200000:2.0:1788572250.301042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c0:0x0](ffff920a50b27448) 00000080:00200000:2.0:1788572250.301052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c1:0x0](ffff920a50b1cb08) 00000080:00200000:2.0:1788572250.301062:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c1:0x0](ffff920a50b1f448) 00000080:00200000:2.0:1788572250.301071:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c2:0x0](ffff920a50b1db88) 00000080:00200000:2.0:1788572250.301081:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c2:0x0](ffff920a50b1aa08) 00000080:00200000:2.0:1788572250.301090:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c3:0x0](ffff920a50b03a88) 00000080:00200000:2.0:1788572250.301100:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c3:0x0](ffff920a50b2e3c8) 00000080:00200000:2.0:1788572250.301109:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c4:0x0](ffff920a50b33a88) 00000080:00200000:2.0:1788572250.301198:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c4:0x0](ffff920a50b30908) 00000080:00200000:2.0:1788572250.301211:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c5:0x0](ffff920a50b2cb08) 00000080:00200000:2.0:1788572250.301221:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c5:0x0](ffff920a50b29988) 00000080:00200000:2.0:1788572250.301231:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c6:0x0](ffff920a50ad9988) 00000080:00200000:2.0:1788572250.301244:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c6:0x0](ffff920a50adcb08) 00000080:00200000:2.0:1788572250.301254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c7:0x0](ffff920a50b2a1c8) 00000080:00200000:2.0:1788572250.301264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c7:0x0](ffff920a50b29148) 00000080:00200000:2.0:1788572250.301273:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c8:0x0](ffff920a50b2ba88) 00000080:00200000:2.0:1788572250.301282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c8:0x0](ffff920a50b28908) 00000080:00200000:2.0:1788572250.301291:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2c9:0x0](ffff920a50b36c08) 00000080:00200000:2.0:1788572250.301301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2c9:0x0](ffff920a50b35348) 00000080:00200000:2.0:1788572250.301311:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ca:0x0](ffff920a50b33248) 00000080:00200000:2.0:1788572250.301320:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ca:0x0](ffff920a50b37448) 00000080:00200000:2.0:1788572250.301330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2cb:0x0](ffff920a50b2db88) 00000080:00200000:2.0:1788572250.301339:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2cb:0x0](ffff920a50b2ec08) 00000080:00200000:2.0:1788572250.301349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2cc:0x0](ffff920a50b363c8) 00000080:00200000:2.0:1788572250.301359:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2cc:0x0](ffff920a50b34b08) 00000080:00200000:2.0:1788572250.301369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2cd:0x0](ffff920a50b2aa08) 00000080:00200000:2.0:1788572250.301436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2cd:0x0](ffff920a50b2d348) 00000080:00200000:2.0:1788572250.301448:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ce:0x0](ffff920a50b31988) 00000080:00200000:2.0:1788572250.301523:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ce:0x0](ffff920a50b321c8) 00000080:00200000:2.0:1788572250.301536:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2cf:0x0](ffff920a50b280c8) 00000080:00200000:2.0:1788572250.301546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2cf:0x0](ffff920a50b2b248) 00000080:00200000:2.0:1788572250.301556:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d0:0x0](ffff920a50b300c8) 00000080:00200000:2.0:1788572250.301567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d0:0x0](ffff920a50b35b88) 00000080:00200000:2.0:1788572250.301579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d1:0x0](ffff920a50b2f448) 00000080:00200000:2.0:1788572250.301676:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d1:0x0](ffff920a50b2c2c8) 00000080:00200000:2.0:1788572250.301688:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d2:0x0](ffff920a50b32a08) 00000080:00200000:2.0:1788572250.301700:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d2:0x0](ffff920a50b342c8) 00000080:00200000:2.0:1788572250.301710:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d3:0x0](ffff920a50b08908) 00000080:00200000:2.0:1788572250.301743:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d3:0x0](ffff920a50b3ec08) 00000080:00200000:2.0:1788572250.301756:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d4:0x0](ffff920a50b3b248) 00000080:00200000:2.0:1788572250.301766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d4:0x0](ffff920a50b3f448) 00000080:00200000:2.0:1788572250.301775:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d5:0x0](ffff920a50b1b248) 00000080:00200000:2.0:1788572250.301785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d5:0x0](ffff920a50b46c08) 00000080:00200000:2.0:1788572250.301793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d6:0x0](ffff920a50b45348) 00000080:00200000:2.0:1788572250.301802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d6:0x0](ffff920a50b43248) 00000080:00200000:2.0:1788572250.301811:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d7:0x0](ffff920a50b3e3c8) 00000080:00200000:2.0:1788572250.301820:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d7:0x0](ffff920a50b3cb08) 00000080:00200000:2.0:1788572250.301829:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d8:0x0](ffff920a50add348) 00000080:00200000:2.0:1788572250.301838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d8:0x0](ffff920a50adba88) 00000080:00200000:2.0:1788572250.301848:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2d9:0x0](ffff920a50b39988) 00000080:00200000:2.0:1788572250.301856:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2d9:0x0](ffff920a50b3a1c8) 00000080:00200000:2.0:1788572250.301866:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2da:0x0](ffff920a50b39148) 00000080:00200000:2.0:1788572250.301875:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2da:0x0](ffff920a50b3ba88) 00000080:00200000:2.0:1788572250.301884:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2db:0x0](ffff920a50b38908) 00000080:00200000:2.0:1788572250.301892:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2db:0x0](ffff920a50b3c2c8) 00000080:00200000:2.0:1788572250.301902:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2dc:0x0](ffff920a50b24b08) 00000080:00200000:2.0:1788572250.301913:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2dc:0x0](ffff920a50b21988) 00000080:00200000:2.0:1788572250.301923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2dd:0x0](ffff920a50b3aa08) 00000080:00200000:2.0:1788572250.301944:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2dd:0x0](ffff920a50b380c8) 00000080:00200000:2.0:1788572250.301955:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2de:0x0](ffff920a50b3db88) 00000080:00200000:2.0:1788572250.301964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2de:0x0](ffff920a50b4b248) 00000080:00200000:2.0:1788572250.301972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2df:0x0](ffff920a50b463c8) 00000080:00200000:2.0:1788572250.301981:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2df:0x0](ffff920a50b44b08) 00000080:00200000:2.0:1788572250.301990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e0:0x0](ffff920a50b41988) 00000080:00200000:2.0:1788572250.301998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e0:0x0](ffff920a50b421c8) 00000080:00200000:2.0:1788572250.302007:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e1:0x0](ffff920a50b480c8) 00000080:00200000:2.0:1788572250.302017:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e1:0x0](ffff920a50b4e3c8) 00000080:00200000:2.0:1788572250.302025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e2:0x0](ffff920a50b4f448) 00000080:00200000:2.0:1788572250.302033:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e2:0x0](ffff920a50b4cb08) 00000080:00200000:2.0:1788572250.302043:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e3:0x0](ffff920a50b49988) 00000080:00200000:2.0:1788572250.302052:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e3:0x0](ffff920a50b4a1c8) 00000080:00200000:2.0:1788572250.302061:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e4:0x0](ffff920a50b442c8) 00000080:00200000:2.0:1788572250.302069:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e4:0x0](ffff920a50b42a08) 00000080:00200000:2.0:1788572250.302077:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e5:0x0](ffff920a50b49148) 00000080:00200000:2.0:1788572250.302087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e5:0x0](ffff920a50b4ba88) 00000080:00200000:2.0:1788572250.302094:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e6:0x0](ffff920a50b48908) 00000080:00200000:2.0:1788572250.302102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e6:0x0](ffff920a50b4c2c8) 00000080:00200000:2.0:1788572250.302112:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e7:0x0](ffff920a50b40908) 00000080:00200000:2.0:1788572250.302307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e7:0x0](ffff920a50b43a88) 00000080:00200000:2.0:1788572250.302317:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e8:0x0](ffff920a50b45b88) 00000080:00200000:2.0:1788572250.302326:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e8:0x0](ffff920a50b41148) 00000080:00200000:2.0:1788572250.302336:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2e9:0x0](ffff920a50b400c8) 00000080:00200000:2.0:1788572250.302346:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2e9:0x0](ffff920a50b47448) 00000080:00200000:2.0:1788572250.302356:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ea:0x0](ffff920a50a980c8) 00000080:00200000:2.0:1788572250.302370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ea:0x0](ffff920a50b52a08) 00000080:00200000:2.0:1788572250.302453:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2eb:0x0](ffff920a50b4ec08) 00000080:00200000:2.0:1788572250.303401:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2eb:0x0](ffff920a50b4d348) 00000080:00200000:2.0:1788572250.303418:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ec:0x0](ffff920a50b5db88) 00000080:00200000:2.0:1788572250.303426:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ec:0x0](ffff920a50b5ec08) 00000080:00200000:2.0:1788572250.303435:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ed:0x0](ffff920a50b4aa08) 00000080:00200000:2.0:1788572250.303444:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ed:0x0](ffff920a50b3d348) 00000080:00200000:2.0:1788572250.303451:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ee:0x0](ffff920a50b23a88) 00000080:00200000:2.0:1788572250.303475:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ee:0x0](ffff920a50b20908) 00000080:00200000:2.0:1788572250.303484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ef:0x0](ffff920a50b63248) 00000080:00200000:2.0:1788572250.303493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ef:0x0](ffff920a50b67448) 00000080:00200000:2.0:1788572250.303502:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f0:0x0](ffff920a50b56c08) 00000080:00200000:2.0:1788572250.303511:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f0:0x0](ffff920a50b55348) 00000080:00200000:2.0:1788572250.303519:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f1:0x0](ffff920a50b53248) 00000080:00200000:2.0:1788572250.303528:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f1:0x0](ffff920a50b57448) 00000080:00200000:2.0:1788572250.303536:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f2:0x0](ffff920a50b5d348) 00000080:00200000:2.0:1788572250.303545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f2:0x0](ffff920a50b5b248) 00000080:00200000:2.0:1788572250.303554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f3:0x0](ffff920a50b22a08) 00000080:00200000:2.0:1788572250.303564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f3:0x0](ffff920a50b25b88) 00000080:00200000:2.0:1788572250.303572:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f4:0x0](ffff920a50b26c08) 00000080:00200000:2.0:1788572250.303581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f4:0x0](ffff920a50b25348) 00000080:00200000:2.0:1788572250.303614:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f5:0x0](ffff920a50b500c8) 00000080:00200000:2.0:1788572250.303624:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f5:0x0](ffff920a50b563c8) 00000080:00200000:2.0:1788572250.303633:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f6:0x0](ffff920a50b580c8) 00000080:00200000:2.0:1788572250.303642:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f6:0x0](ffff920a50b5e3c8) 00000080:00200000:2.0:1788572250.303651:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f7:0x0](ffff920a50b663c8) 00000080:00200000:2.0:1788572250.303660:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f7:0x0](ffff920a50b64b08) 00000080:00200000:2.0:1788572250.303669:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f8:0x0](ffff920a50b59988) 00000080:00200000:2.0:1788572250.303678:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f8:0x0](ffff920a50b5a1c8) 00000080:00200000:2.0:1788572250.303687:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2f9:0x0](ffff920a50b621c8) 00000080:00200000:2.0:1788572250.303697:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2f9:0x0](ffff920a50b61148) 00000080:00200000:2.0:1788572250.303706:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2fa:0x0](ffff920a50b51988) 00000080:00200000:2.0:1788572250.303715:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2fa:0x0](ffff920a50b521c8) 00000080:00200000:2.0:1788572250.303723:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2fb:0x0](ffff920a50b59148) 00000080:00200000:2.0:1788572250.303731:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2fb:0x0](ffff920a50b5ba88) 00000080:00200000:2.0:1788572250.303739:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2fc:0x0](ffff920a50b58908) 00000080:00200000:2.0:1788572250.303748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2fc:0x0](ffff920a50b5c2c8) 00000080:00200000:2.0:1788572250.303757:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2fd:0x0](ffff920a50b60908) 00000080:00200000:2.0:1788572250.303765:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2fd:0x0](ffff920a50b642c8) 00000080:00200000:2.0:1788572250.303774:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2fe:0x0](ffff920a50b5aa08) 00000080:00200000:2.0:1788572250.303781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2fe:0x0](ffff920a50b5cb08) 00000080:00200000:2.0:1788572250.303790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x2ff:0x0](ffff920a50b5f448) 00000080:00200000:2.0:1788572250.303798:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x2ff:0x0](ffff920a50b4db88) 00000080:00200000:2.0:1788572250.303806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x300:0x0](ffff920a50b62a08) 00000080:00200000:2.0:1788572250.303814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x300:0x0](ffff920a50b63a88) 00000080:00200000:2.0:1788572250.303822:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x301:0x0](ffff920a50b65b88) 00000080:00200000:2.0:1788572250.303830:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x301:0x0](ffff920a50b61988) 00000080:00200000:2.0:1788572250.303838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x302:0x0](ffff920a50b600c8) 00000080:00200000:2.0:1788572250.303846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x302:0x0](ffff920a50b65348) 00000080:00200000:2.0:1788572250.303854:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x303:0x0](ffff920a50b31148) 00000080:00200000:2.0:1788572250.303862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x303:0x0](ffff920a50b6c2c8) 00000080:00200000:2.0:1788572250.303872:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x304:0x0](ffff920a50b66c08) 00000080:00200000:2.0:1788572250.303880:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x304:0x0](ffff920a50b75b88) 00000080:00200000:2.0:1788572250.303888:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x305:0x0](ffff920a50b76c08) 00000080:00200000:2.0:1788572250.303896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x305:0x0](ffff920a50b75348) 00000080:00200000:2.0:1788572250.303904:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x306:0x0](ffff920a50b542c8) 00000080:00200000:2.0:1788572250.303912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x306:0x0](ffff920a50b50908) 00000080:00200000:2.0:1788572250.303920:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x307:0x0](ffff920a50b77448) 00000080:00200000:2.0:1788572250.303941:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x307:0x0](ffff920a50b700c8) 00000080:00200000:2.0:1788572250.303951:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x308:0x0](ffff920a50b53a88) 00000080:00200000:2.0:1788572250.303959:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x308:0x0](ffff920a50b51148) 00000080:00200000:2.0:1788572250.303968:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x309:0x0](ffff920a50b73248) 00000080:00200000:2.0:1788572250.303978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x309:0x0](ffff920a50b763c8) 00000080:00200000:2.0:1788572250.303985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30a:0x0](ffff920a50b54b08) 00000080:00200000:2.0:1788572250.303994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30a:0x0](ffff920a50b55b88) 00000080:00200000:2.0:1788572250.304002:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30b:0x0](ffff920a50b71988) 00000080:00200000:2.0:1788572250.304010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30b:0x0](ffff920a50b721c8) 00000080:00200000:2.0:1788572250.304019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30c:0x0](ffff920a50b79148) 00000080:00200000:2.0:1788572250.304028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30c:0x0](ffff920a50b7ba88) 00000080:00200000:2.0:1788572250.304037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30d:0x0](ffff920a50b73a88) 00000080:00200000:2.0:1788572250.304045:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30d:0x0](ffff920a50b70908) 00000080:00200000:2.0:1788572250.304054:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30e:0x0](ffff920a50b6f448) 00000080:00200000:2.0:1788572250.304063:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30e:0x0](ffff920a50b680c8) 00000080:00200000:2.0:1788572250.304072:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x30f:0x0](ffff920a50b71148) 00000080:00200000:2.0:1788572250.304080:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x30f:0x0](ffff920a50b742c8) 00000080:00200000:2.0:1788572250.304088:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x310:0x0](ffff920a50b263c8) 00000080:00200000:2.0:1788572250.304102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x310:0x0](ffff920a50b200c8) 00000080:00200000:2.0:1788572250.304112:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x311:0x0](ffff920a50b6cb08) 00000080:00200000:2.0:1788572250.304317:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x311:0x0](ffff920a50b69988) 00000080:00200000:2.0:1788572250.304328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x312:0x0](ffff920a50b74b08) 00000080:00200000:2.0:1788572250.304338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x312:0x0](ffff920a50b842c8) 00000080:00200000:2.0:1788572250.304349:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x313:0x0](ffff920a50b7c2c8) 00000080:00200000:2.0:1788572250.304359:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x313:0x0](ffff920a50b7aa08) 00000080:00200000:2.0:1788572250.304367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x314:0x0](ffff920a50b85b88) 00000080:00200000:2.0:1788572250.304377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x314:0x0](ffff920a50b86c08) 00000080:00200000:2.0:1788572250.304387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x315:0x0](ffff920a50b85348) 00000080:00200000:2.0:1788572250.304396:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x315:0x0](ffff920a50b83248) 00000080:00200000:2.0:1788572250.304406:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x316:0x0](ffff920a50b221c8) 00000080:00200000:2.0:1788572250.304417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x316:0x0](ffff920a50b21148) 00000080:00200000:2.0:1788572250.304426:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x317:0x0](ffff920a50b69148) 00000080:00200000:2.0:1788572250.304436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x317:0x0](ffff920a50b6ba88) 00000080:00200000:2.0:1788572250.304446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x318:0x0](ffff920a50b800c8) 00000080:00200000:2.0:1788572250.304454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x318:0x0](ffff920a50b863c8) 00000080:00200000:2.0:1788572250.304464:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x319:0x0](ffff920a50b6a1c8) 00000080:00200000:2.0:1788572250.304474:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x319:0x0](ffff920a50b6e3c8) 00000080:00200000:2.0:1788572250.304483:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31a:0x0](ffff920a50b84b08) 00000080:00200000:2.0:1788572250.304492:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31a:0x0](ffff920a50b81988) 00000080:00200000:2.0:1788572250.304501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31b:0x0](ffff920a50b821c8) 00000080:00200000:2.0:1788572250.304510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31b:0x0](ffff920a50b81148) 00000080:00200000:2.0:1788572250.304520:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31c:0x0](ffff920a50b6d348) 00000080:00200000:2.0:1788572250.304531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31c:0x0](ffff920a50b6ec08) 00000080:00200000:2.0:1788572250.304540:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31d:0x0](ffff920a50b83a88) 00000080:00200000:2.0:1788572250.304548:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31d:0x0](ffff920a50b80908) 00000080:00200000:2.0:1788572250.304557:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31e:0x0](ffff920a50b87448) 00000080:00200000:2.0:1788572250.304567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31e:0x0](ffff920a50b82a08) 00000080:00200000:2.0:1788572250.304575:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x31f:0x0](ffff920a50b242c8) 00000080:00200000:2.0:1788572250.304608:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x31f:0x0](ffff920a50b8ba88) 00000080:00200000:2.0:1788572250.304619:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x320:0x0](ffff920a50b72a08) 00000080:00200000:2.0:1788572250.304628:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x320:0x0](ffff920a50b91148) 00000080:00200000:2.0:1788572250.304638:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x321:0x0](ffff920a50b93a88) 00000080:00200000:2.0:1788572250.304648:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x321:0x0](ffff920a50b90908) 00000080:00200000:2.0:1788572250.304656:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x322:0x0](ffff920a50b6aa08) 00000080:00200000:2.0:1788572250.304665:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x322:0x0](ffff920a50b6db88) 00000080:00200000:2.0:1788572250.304673:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x323:0x0](ffff920a50b942c8) 00000080:00200000:2.0:1788572250.304682:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x323:0x0](ffff920a50b92a08) 00000080:00200000:2.0:1788572250.304691:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x324:0x0](ffff920a50b68908) 00000080:00200000:2.0:1788572250.304701:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x324:0x0](ffff920a50b9aa08) 00000080:00200000:2.0:1788572250.304711:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x325:0x0](ffff920a50b8c2c8) 00000080:00200000:2.0:1788572250.304720:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x325:0x0](ffff920a50b8aa08) 00000080:00200000:2.0:1788572250.304728:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x326:0x0](ffff920a50b9db88) 00000080:00200000:2.0:1788572250.304737:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x326:0x0](ffff920a50b9ec08) 00000080:00200000:2.0:1788572250.304747:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x327:0x0](ffff920a50b9b248) 00000080:00200000:2.0:1788572250.304757:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x327:0x0](ffff920a50b9f448) 00000080:00200000:2.0:1788572250.304766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x328:0x0](ffff920a50b7f448) 00000080:00200000:2.0:1788572250.304775:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x328:0x0](ffff920a50b780c8) 00000080:00200000:2.0:1788572250.304783:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x329:0x0](ffff920a50b8db88) 00000080:00200000:2.0:1788572250.304792:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x329:0x0](ffff920a50b8ec08) 00000080:00200000:2.0:1788572250.304801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32a:0x0](ffff920a50b8d348) 00000080:00200000:2.0:1788572250.304810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32a:0x0](ffff920a50b8b248) 00000080:00200000:2.0:1788572250.304819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32b:0x0](ffff920a50b980c8) 00000080:00200000:2.0:1788572250.304831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32b:0x0](ffff920a50b9e3c8) 00000080:00200000:2.0:1788572250.304840:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32c:0x0](ffff920a50b8f448) 00000080:00200000:2.0:1788572250.304849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32c:0x0](ffff920a50b880c8) 00000080:00200000:2.0:1788572250.304858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32d:0x0](ffff920a50b900c8) 00000080:00200000:2.0:1788572250.304867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32d:0x0](ffff920a50b963c8) 00000080:00200000:2.0:1788572250.304876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32e:0x0](ffff920a50b7e3c8) 00000080:00200000:2.0:1788572250.304886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32e:0x0](ffff920a50b7cb08) 00000080:00200000:2.0:1788572250.304894:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x32f:0x0](ffff920a50b99988) 00000080:00200000:2.0:1788572250.304903:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x32f:0x0](ffff920a50b9a1c8) 00000080:00200000:2.0:1788572250.304912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x330:0x0](ffff920a50b94b08) 00000080:00200000:2.0:1788572250.304923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x330:0x0](ffff920a50b91988) 00000080:00200000:2.0:1788572250.304949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x331:0x0](ffff920a50b9ba88) 00000080:00200000:2.0:1788572250.304958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x331:0x0](ffff920a50b98908) 00000080:00200000:2.0:1788572250.304967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x332:0x0](ffff920a50b79988) 00000080:00200000:2.0:1788572250.304975:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x332:0x0](ffff920a50b7a1c8) 00000080:00200000:2.0:1788572250.304985:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x333:0x0](ffff920a50b921c8) 00000080:00200000:2.0:1788572250.304993:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x333:0x0](ffff920a50b97448) 00000080:00200000:2.0:1788572250.305001:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x334:0x0](ffff920a50b8cb08) 00000080:00200000:2.0:1788572250.305010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x334:0x0](ffff920a50b89988) 00000080:00200000:2.0:1788572250.305019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x335:0x0](ffff920a50b9c2c8) 00000080:00200000:2.0:1788572250.305028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x335:0x0](ffff920a50b99148) 00000080:00200000:2.0:1788572250.305036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x336:0x0](ffff920a50b89148) 00000080:00200000:2.0:1788572250.305053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x336:0x0](ffff920a50b8a1c8) 00000080:00200000:2.0:1788572250.305062:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x337:0x0](ffff920a50b96c08) 00000080:00200000:2.0:1788572250.305071:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x337:0x0](ffff920a50b95b88) 00000080:00200000:2.0:1788572250.305108:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x338:0x0](ffff920a50b9d348) 00000080:00200000:2.0:1788572250.305352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x338:0x0](ffff920a50b9cb08) 00000080:00200000:2.0:1788572250.305364:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x339:0x0](ffff920a50b93248) 00000080:00200000:2.0:1788572250.305381:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x339:0x0](ffff920a50ba00c8) 00000080:00200000:2.0:1788572250.305391:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33a:0x0](ffff920a50ba63c8) 00000080:00200000:2.0:1788572250.305401:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33a:0x0](ffff920a50ba4b08) 00000080:00200000:2.0:1788572250.305409:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33b:0x0](ffff920a50b95348) 00000080:00200000:2.0:1788572250.305418:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33b:0x0](ffff920a50b6b248) 00000080:00200000:2.0:1788572250.305427:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33c:0x0](ffff920a50b88908) 00000080:00200000:2.0:1788572250.305436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33c:0x0](ffff920a50b8e3c8) 00000080:00200000:2.0:1788572250.305447:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33d:0x0](ffff920a50ba9148) 00000080:00200000:2.0:1788572250.305457:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33d:0x0](ffff920a50baba88) 00000080:00200000:2.0:1788572250.305466:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33e:0x0](ffff920a50bb7448) 00000080:00200000:2.0:1788572250.305475:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33e:0x0](ffff920a50bb00c8) 00000080:00200000:2.0:1788572250.305484:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x33f:0x0](ffff920a50ba8908) 00000080:00200000:2.0:1788572250.305493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x33f:0x0](ffff920a50bac2c8) 00000080:00200000:2.0:1788572250.305501:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x340:0x0](ffff920a50baa1c8) 00000080:00200000:2.0:1788572250.305510:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x340:0x0](ffff920a50baaa08) 00000080:00200000:2.0:1788572250.305520:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x341:0x0](ffff920a50ba3a88) 00000080:00200000:2.0:1788572250.305529:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x341:0x0](ffff920a50ba0908) 00000080:00200000:2.0:1788572250.305537:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x342:0x0](ffff920a50bad348) 00000080:00200000:2.0:1788572250.305546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x342:0x0](ffff920a50bab248) 00000080:00200000:2.0:1788572250.305554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x343:0x0](ffff920a50ba2a08) 00000080:00200000:2.0:1788572250.305563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x343:0x0](ffff920a50ba5b88) 00000080:00200000:2.0:1788572250.305571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x344:0x0](ffff920a50ba6c08) 00000080:00200000:2.0:1788572250.305579:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x344:0x0](ffff920a50ba5348) 00000080:00200000:2.0:1788572250.305771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x345:0x0](ffff920a50ba7448) 00000080:00200000:2.0:1788572250.305785:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x345:0x0](ffff920a50ba42c8) 00000080:00200000:2.0:1788572250.305795:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x346:0x0](ffff920a50baf448) 00000080:00200000:2.0:1788572250.305804:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x346:0x0](ffff920a50ba80c8) 00000080:00200000:2.0:1788572250.305814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x347:0x0](ffff920a50ba1148) 00000080:00200000:2.0:1788572250.305823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x347:0x0](ffff920a50ba21c8) 00000080:00200000:2.0:1788572250.305831:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x348:0x0](ffff920a50ba3248) 00000080:00200000:2.0:1788572250.305839:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x348:0x0](ffff920a50bbdb88) 00000080:00200000:2.0:1788572250.305849:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x349:0x0](ffff920a50bbec08) 00000080:00200000:2.0:1788572250.305858:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x349:0x0](ffff920a50bbd348) 00000080:00200000:2.0:1788572250.305868:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34a:0x0](ffff920a50bbb248) 00000080:00200000:2.0:1788572250.305877:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34a:0x0](ffff920a50bbf448) 00000080:00200000:2.0:1788572250.305886:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34b:0x0](ffff920a50bbe3c8) 00000080:00200000:2.0:1788572250.305896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34b:0x0](ffff920a50bbcb08) 00000080:00200000:2.0:1788572250.305905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34c:0x0](ffff920a50bb63c8) 00000080:00200000:2.0:1788572250.305914:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34c:0x0](ffff920a50bb4b08) 00000080:00200000:2.0:1788572250.305923:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34d:0x0](ffff920a50b7ec08) 00000080:00200000:2.0:1788572250.305943:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34d:0x0](ffff920a50b7db88) 00000080:00200000:2.0:1788572250.305952:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34e:0x0](ffff920a50bb9988) 00000080:00200000:2.0:1788572250.305962:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34e:0x0](ffff920a50bba1c8) 00000080:00200000:2.0:1788572250.305972:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x34f:0x0](ffff920a50b7d348) 00000080:00200000:2.0:1788572250.305982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x34f:0x0](ffff920a50b7b248) 00000080:00200000:2.0:1788572250.305990:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x350:0x0](ffff920a50bacb08) 00000080:00200000:2.0:1788572250.305999:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x350:0x0](ffff920a50bae3c8) 00000080:00200000:2.0:1788572250.306008:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x351:0x0](ffff920a50bbba88) 00000080:00200000:2.0:1788572250.306039:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x351:0x0](ffff920a50bb8908) 00000080:00200000:2.0:1788572250.306056:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x352:0x0](ffff920a50bbc2c8) 00000080:00200000:2.0:1788572250.306068:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x352:0x0](ffff920a50bbaa08) 00000080:00200000:2.0:1788572250.306078:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x353:0x0](ffff920a50baec08) 00000080:00200000:2.0:1788572250.306087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x353:0x0](ffff920a50badb88) 00000080:00200000:2.0:1788572250.306095:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x354:0x0](ffff920a50bb21c8) 00000080:00200000:2.0:1788572250.306105:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x354:0x0](ffff920a50bb1148) 00000080:00200000:2.0:1788572250.306223:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x355:0x0](ffff920a50bb1988) 00000080:00200000:2.0:1788572250.306238:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x355:0x0](ffff920a50bb3a88) 00000080:00200000:2.0:1788572250.306248:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x356:0x0](ffff920a50bc6c08) 00000080:00200000:2.0:1788572250.306258:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x356:0x0](ffff920a50bc5348) 00000080:00200000:2.0:1788572250.306267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x357:0x0](ffff920a50bb9148) 00000080:00200000:2.0:1788572250.306276:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x357:0x0](ffff920a50ba1988) 00000080:00200000:2.0:1788572250.306285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x358:0x0](ffff920a50bc9988) 00000080:00200000:2.0:1788572250.306295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x358:0x0](ffff920a50bca1c8) 00000080:00200000:2.0:1788572250.306304:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x359:0x0](ffff920a50bd0908) 00000080:00200000:2.0:1788572250.306315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x359:0x0](ffff920a50bd42c8) 00000080:00200000:2.0:1788572250.306324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35a:0x0](ffff920a50bc00c8) 00000080:00200000:2.0:1788572250.306333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35a:0x0](ffff920a50bc63c8) 00000080:00200000:2.0:1788572250.306341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35b:0x0](ffff920a50bc4b08) 00000080:00200000:2.0:1788572250.306350:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35b:0x0](ffff920a50bc1988) 00000080:00200000:2.0:1788572250.306361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35c:0x0](ffff920a50bc8908) 00000080:00200000:2.0:1788572250.306369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35c:0x0](ffff920a50bcc2c8) 00000080:00200000:2.0:1788572250.306378:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35d:0x0](ffff920a50bc1148) 00000080:00200000:2.0:1788572250.306387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35d:0x0](ffff920a50bc3a88) 00000080:00200000:2.0:1788572250.306397:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35e:0x0](ffff920a50bb0908) 00000080:00200000:2.0:1788572250.306406:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35e:0x0](ffff920a50bb42c8) 00000080:00200000:2.0:1788572250.306415:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x35f:0x0](ffff920a50bb2a08) 00000080:00200000:2.0:1788572250.306424:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x35f:0x0](ffff920a50bb5b88) 00000080:00200000:2.0:1788572250.306432:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x360:0x0](ffff920a50bc21c8) 00000080:00200000:2.0:1788572250.306441:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x360:0x0](ffff920a50bc0908) 00000080:00200000:2.0:1788572250.306450:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x361:0x0](ffff920a50bc42c8) 00000080:00200000:2.0:1788572250.306459:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x361:0x0](ffff920a50bc2a08) 00000080:00200000:2.0:1788572250.306468:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x362:0x0](ffff920a50bcd348) 00000080:00200000:2.0:1788572250.306476:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x362:0x0](ffff920a50bcb248) 00000080:00200000:2.0:1788572250.306485:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x363:0x0](ffff920a50bc7448) 00000080:00200000:2.0:1788572250.306493:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x363:0x0](ffff920a50bc3248) 00000080:00200000:2.0:1788572250.306503:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x364:0x0](ffff920a50bd5b88) 00000080:00200000:2.0:1788572250.306513:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x364:0x0](ffff920a50bd6c08) 00000080:00200000:2.0:1788572250.306521:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x365:0x0](ffff920a50bc5b88) 00000080:00200000:2.0:1788572250.306531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x365:0x0](ffff920a50b78908) 00000080:00200000:2.0:1788572250.306541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x366:0x0](ffff920a50ba9988) 00000080:00200000:2.0:1788572250.306549:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x366:0x0](ffff920a50bdd348) 00000080:00200000:2.0:1788572250.306559:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x367:0x0](ffff920a50bc80c8) 00000080:00200000:2.0:1788572250.306568:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x367:0x0](ffff920a50bce3c8) 00000080:00200000:2.0:1788572250.306576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x368:0x0](ffff920a50bdf448) 00000080:00200000:2.0:1788572250.306901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x368:0x0](ffff920a50bd80c8) 00000080:00200000:2.0:1788572250.306917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x369:0x0](ffff920a50bccb08) 00000080:00200000:2.0:1788572250.306937:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x369:0x0](ffff920a50bcf448) 00000080:00200000:2.0:1788572250.306947:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36a:0x0](ffff920a50bb3248) 00000080:00200000:2.0:1788572250.306957:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36a:0x0](ffff920a50bb6c08) 00000080:00200000:2.0:1788572250.306986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36b:0x0](ffff920a50bde3c8) 00000080:00200000:2.0:1788572250.307000:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36b:0x0](ffff920a50bdcb08) 00000080:00200000:2.0:1788572250.307010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36c:0x0](ffff920a50bcdb88) 00000080:00200000:2.0:1788572250.307019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36c:0x0](ffff920a50bcaa08) 00000080:00200000:2.0:1788572250.307028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36d:0x0](ffff920a50bd00c8) 00000080:00200000:2.0:1788572250.307038:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36d:0x0](ffff920a50bd63c8) 00000080:00200000:2.0:1788572250.307048:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36e:0x0](ffff920a50bd9988) 00000080:00200000:2.0:1788572250.307057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36e:0x0](ffff920a50bda1c8) 00000080:00200000:2.0:1788572250.307066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x36f:0x0](ffff920a50bc9148) 00000080:00200000:2.0:1788572250.307076:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x36f:0x0](ffff920a50bcba88) 00000080:00200000:2.0:1788572250.307085:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x370:0x0](ffff920a50bdba88) 00000080:00200000:2.0:1788572250.307098:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x370:0x0](ffff920a50bd8908) 00000080:00200000:2.0:1788572250.307106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x371:0x0](ffff920a50bb5348) 00000080:00200000:2.0:1788572250.307230:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x371:0x0](ffff920a50bb80c8) 00000080:00200000:2.0:1788572250.307244:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x372:0x0](ffff920a50bdc2c8) 00000080:00200000:2.0:1788572250.307254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x372:0x0](ffff920a50bdaa08) 00000080:00200000:2.0:1788572250.307264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x373:0x0](ffff920a50bd4b08) 00000080:00200000:2.0:1788572250.307274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x373:0x0](ffff920a50bd1988) 00000080:00200000:2.0:1788572250.307285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x374:0x0](ffff920a50bcec08) 00000080:00200000:2.0:1788572250.307295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x374:0x0](ffff920a50be2a08) 00000080:00200000:2.0:1788572250.307304:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x375:0x0](ffff920a50bd21c8) 00000080:00200000:2.0:1788572250.307314:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x375:0x0](ffff920a50bd1148) 00000080:00200000:2.0:1788572250.307324:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x376:0x0](ffff920a50be8908) 00000080:00200000:2.0:1788572250.307334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x376:0x0](ffff920a50bec2c8) 00000080:00200000:2.0:1788572250.307343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x377:0x0](ffff920a50be5b88) 00000080:00200000:2.0:1788572250.307352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x377:0x0](ffff920a50be6c08) 00000080:00200000:2.0:1788572250.307362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x378:0x0](ffff920a50bddb88) 00000080:00200000:2.0:1788572250.307371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x378:0x0](ffff920a50bd9148) 00000080:00200000:2.0:1788572250.307380:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x379:0x0](ffff920a50be5348) 00000080:00200000:2.0:1788572250.307390:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x379:0x0](ffff920a50be3248) 00000080:00200000:2.0:1788572250.307400:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37a:0x0](ffff920a50bdb248) 00000080:00200000:2.0:1788572250.307408:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37a:0x0](ffff920a50bdec08) 00000080:00200000:2.0:1788572250.307417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37b:0x0](ffff920a50be63c8) 00000080:00200000:2.0:1788572250.307427:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37b:0x0](ffff920a50be4b08) 00000080:00200000:2.0:1788572250.307437:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37c:0x0](ffff920a50bf0908) 00000080:00200000:2.0:1788572250.307446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37c:0x0](ffff920a50bf42c8) 00000080:00200000:2.0:1788572250.307456:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37d:0x0](ffff920a50bf2a08) 00000080:00200000:2.0:1788572250.307466:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37d:0x0](ffff920a50bf5b88) 00000080:00200000:2.0:1788572250.307476:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37e:0x0](ffff920a50be1988) 00000080:00200000:2.0:1788572250.307485:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37e:0x0](ffff920a50be21c8) 00000080:00200000:2.0:1788572250.307495:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x37f:0x0](ffff920a50be1148) 00000080:00200000:2.0:1788572250.307504:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x37f:0x0](ffff920a50be3a88) 00000080:00200000:2.0:1788572250.307515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x380:0x0](ffff920a50bf7448) 00000080:00200000:2.0:1788572250.307524:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x380:0x0](ffff920a50bf00c8) 00000080:00200000:2.0:1788572250.307534:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x381:0x0](ffff920a50beaa08) 00000080:00200000:2.0:1788572250.307545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x381:0x0](ffff920a50bedb88) 00000080:00200000:2.0:1788572250.307553:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x382:0x0](ffff920a50bf63c8) 00000080:00200000:2.0:1788572250.307561:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x382:0x0](ffff920a50bf4b08) 00000080:00200000:2.0:1788572250.307569:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x383:0x0](ffff920a50beec08) 00000080:00200000:2.0:1788572250.307576:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x383:0x0](ffff920a50bed348) 00000080:00200000:2.0:1788572250.307617:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x384:0x0](ffff920a50be42c8) 00000080:00200000:2.0:1788572250.307644:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x384:0x0](ffff920a50be0908) 00000080:00200000:2.0:1788572250.307656:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x385:0x0](ffff920a50bf21c8) 00000080:00200000:2.0:1788572250.307665:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x385:0x0](ffff920a50bf1148) 00000080:00200000:2.0:1788572250.307674:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x386:0x0](ffff920a50be7448) 00000080:00200000:2.0:1788572250.307684:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x386:0x0](ffff920a50bfec08) 00000080:00200000:2.0:1788572250.307693:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x387:0x0](ffff920a50bfb248) 00000080:00200000:2.0:1788572250.307703:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x387:0x0](ffff920a50bff448) 00000080:00200000:2.0:1788572250.307713:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x388:0x0](ffff920a50bf80c8) 00000080:00200000:2.0:1788572250.307721:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x388:0x0](ffff920a50bfe3c8) 00000080:00200000:2.0:1788572250.307731:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x389:0x0](ffff920a50bf9988) 00000080:00200000:2.0:1788572250.307740:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x389:0x0](ffff920a50bfa1c8) 00000080:00200000:2.0:1788572250.307749:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38a:0x0](ffff920a50bf3248) 00000080:00200000:2.0:1788572250.307757:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38a:0x0](ffff920a50bf5348) 00000080:00200000:2.0:1788572250.307766:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38b:0x0](ffff920a50bd3a88) 00000080:00200000:2.0:1788572250.307777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38b:0x0](ffff920a50bd7448) 00000080:00200000:2.0:1788572250.307786:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38c:0x0](ffff920a50bf6c08) 00000080:00200000:2.0:1788572250.307793:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38c:0x0](ffff920a50bf3a88) 00000080:00200000:2.0:1788572250.307801:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38d:0x0](ffff920a50bee3c8) 00000080:00200000:2.0:1788572250.307810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38d:0x0](ffff920a50becb08) 00000080:00200000:2.0:1788572250.307819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38e:0x0](ffff920a50bf9148) 00000080:00200000:2.0:1788572250.307828:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38e:0x0](ffff920a50bfba88) 00000080:00200000:2.0:1788572250.307837:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x38f:0x0](ffff920a50c021c8) 00000080:00200000:2.0:1788572250.307846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x38f:0x0](ffff920a50c01148) 00000080:00200000:2.0:1788572250.307856:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x390:0x0](ffff920a50bf8908) 00000080:00200000:2.0:1788572250.307864:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x390:0x0](ffff920a50bfc2c8) 00000080:00200000:2.0:1788572250.307873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x391:0x0](ffff920a50c00908) 00000080:00200000:2.0:1788572250.307882:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x391:0x0](ffff920a50c042c8) 00000080:00200000:2.0:1788572250.307890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x392:0x0](ffff920a50bd3248) 00000080:00200000:2.0:1788572250.307900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x392:0x0](ffff920a50bd5348) 00000080:00200000:2.0:1788572250.307910:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x393:0x0](ffff920a50bd2a08) 00000080:00200000:2.0:1788572250.307919:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x393:0x0](ffff920a50c0d348) 00000080:00200000:2.0:1788572250.307941:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x394:0x0](ffff920a50c05b88) 00000080:00200000:2.0:1788572250.307953:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x394:0x0](ffff920a50c06c08) 00000080:00200000:2.0:1788572250.307963:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x395:0x0](ffff920a50c05348) 00000080:00200000:2.0:1788572250.307971:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x395:0x0](ffff920a50c03248) 00000080:00200000:2.0:1788572250.307979:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x396:0x0](ffff920a50bfdb88) 00000080:00200000:2.0:1788572250.307989:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x396:0x0](ffff920a50bfaa08) 00000080:00200000:2.0:1788572250.307998:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x397:0x0](ffff920a50c000c8) 00000080:00200000:2.0:1788572250.308006:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x397:0x0](ffff920a50c063c8) 00000080:00200000:2.0:1788572250.308015:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x398:0x0](ffff920a50c04b08) 00000080:00200000:2.0:1788572250.308024:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x398:0x0](ffff920a50c01988) 00000080:00200000:2.0:1788572250.308033:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x399:0x0](ffff920a50bfd348) 00000080:00200000:2.0:1788572250.308042:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x399:0x0](ffff920a50bf1988) 00000080:00200000:2.0:1788572250.308051:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39a:0x0](ffff920a50c0f448) 00000080:00200000:2.0:1788572250.308060:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39a:0x0](ffff920a50c080c8) 00000080:00200000:2.0:1788572250.308069:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39b:0x0](ffff920a50be00c8) 00000080:00200000:2.0:1788572250.308079:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39b:0x0](ffff920a50c14b08) 00000080:00200000:2.0:1788572250.308087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39c:0x0](ffff920a50be9148) 00000080:00200000:2.0:1788572250.308098:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39c:0x0](ffff920a50beba88) 00000080:00200000:2.0:1788572250.308107:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39d:0x0](ffff920a50c121c8) 00000080:00200000:2.0:1788572250.308187:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39d:0x0](ffff920a50c11148) 00000080:00200000:2.0:1788572250.308213:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39e:0x0](ffff920a50c03a88) 00000080:00200000:2.0:1788572250.308225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39e:0x0](ffff920a50c07448) 00000080:00200000:2.0:1788572250.308234:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x39f:0x0](ffff920a50bfcb08) 00000080:00200000:2.0:1788572250.308242:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x39f:0x0](ffff920a50c1cb08) 00000080:00200000:2.0:1788572250.308250:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a0:0x0](ffff920a50c19988) 00000080:00200000:2.0:1788572250.308258:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a0:0x0](ffff920a50c1a1c8) 00000080:00200000:2.0:1788572250.308267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a1:0x0](ffff920a50bea1c8) 00000080:00200000:2.0:1788572250.308277:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a1:0x0](ffff920a50be9988) 00000080:00200000:2.0:1788572250.308285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a2:0x0](ffff920a50c1ba88) 00000080:00200000:2.0:1788572250.308295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a2:0x0](ffff920a50c18908) 00000080:00200000:2.0:1788572250.308303:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a3:0x0](ffff920a50c1c2c8) 00000080:00200000:2.0:1788572250.308312:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a3:0x0](ffff920a50c1aa08) 00000080:00200000:2.0:1788572250.308321:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a4:0x0](ffff920a50c1db88) 00000080:00200000:2.0:1788572250.308329:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a4:0x0](ffff920a50c1ec08) 00000080:00200000:2.0:1788572250.308338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a5:0x0](ffff920a50c13a88) 00000080:00200000:2.0:1788572250.308346:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a5:0x0](ffff920a50c10908) 00000080:00200000:2.0:1788572250.308354:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a6:0x0](ffff920a50c142c8) 00000080:00200000:2.0:1788572250.308362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a6:0x0](ffff920a50c12a08) 00000080:00200000:2.0:1788572250.308370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a7:0x0](ffff920a50c15b88) 00000080:00200000:2.0:1788572250.308380:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a7:0x0](ffff920a50c16c08) 00000080:00200000:2.0:1788572250.308389:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a8:0x0](ffff920a50c15348) 00000080:00200000:2.0:1788572250.308397:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a8:0x0](ffff920a50c13248) 00000080:00200000:2.0:1788572250.308407:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3a9:0x0](ffff920a50c09988) 00000080:00200000:2.0:1788572250.308417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3a9:0x0](ffff920a50c0a1c8) 00000080:00200000:2.0:1788572250.308426:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3aa:0x0](ffff920a50c1e3c8) 00000080:00200000:2.0:1788572250.308436:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3aa:0x0](ffff920a50c1f448) 00000080:00200000:2.0:1788572250.308445:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ab:0x0](ffff920a50c1d348) 00000080:00200000:2.0:1788572250.308454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ab:0x0](ffff920a50c19148) 00000080:00200000:2.0:1788572250.308463:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ac:0x0](ffff920a50be80c8) 00000080:00200000:2.0:1788572250.308473:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ac:0x0](ffff920a50bef448) 00000080:00200000:2.0:1788572250.308482:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ad:0x0](ffff920a50c17448) 00000080:00200000:2.0:1788572250.308491:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ad:0x0](ffff920a50c100c8) 00000080:00200000:2.0:1788572250.308499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ae:0x0](ffff920a50c163c8) 00000080:00200000:2.0:1788572250.308508:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ae:0x0](ffff920a50c11988) 00000080:00200000:2.0:1788572250.308517:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3af:0x0](ffff920a50c1b248) 00000080:00200000:2.0:1788572250.308526:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3af:0x0](ffff920a50c180c8) 00000080:00200000:2.0:1788572250.308535:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b0:0x0](ffff920a50c221c8) 00000080:00200000:2.0:1788572250.308546:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b0:0x0](ffff920a50c21148) 00000080:00200000:2.0:1788572250.308554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b1:0x0](ffff920a50c2aa08) 00000080:00200000:2.0:1788572250.308563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b1:0x0](ffff920a50c2db88) 00000080:00200000:2.0:1788572250.308572:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b2:0x0](ffff920a50beb248) 00000080:00200000:2.0:1788572250.308582:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b2:0x0](ffff920a50c02a08) 00000080:00200000:2.0:1788572250.308618:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b3:0x0](ffff920a50c08908) 00000080:00200000:2.0:1788572250.308629:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b3:0x0](ffff920a50c0c2c8) 00000080:00200000:2.0:1788572250.308639:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b4:0x0](ffff920a50c32a08) 00000080:00200000:2.0:1788572250.308649:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b4:0x0](ffff920a50c35b88) 00000080:00200000:2.0:1788572250.308658:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b5:0x0](ffff920a50c2d348) 00000080:00200000:2.0:1788572250.308667:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b5:0x0](ffff920a50c2b248) 00000080:00200000:2.0:1788572250.308676:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b6:0x0](ffff920a50c36c08) 00000080:00200000:2.0:1788572250.308686:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b6:0x0](ffff920a50c35348) 00000080:00200000:2.0:1788572250.308695:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b7:0x0](ffff920a50c0aa08) 00000080:00200000:2.0:1788572250.308726:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b7:0x0](ffff920a50c0db88) 00000080:00200000:2.0:1788572250.308740:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b8:0x0](ffff920a50c280c8) 00000080:00200000:2.0:1788572250.308750:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b8:0x0](ffff920a50c2e3c8) 00000080:00200000:2.0:1788572250.308758:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3b9:0x0](ffff920a50c2cb08) 00000080:00200000:2.0:1788572250.308768:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3b9:0x0](ffff920a50c29988) 00000080:00200000:2.0:1788572250.308777:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ba:0x0](ffff920a50c37448) 00000080:00200000:2.0:1788572250.308787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ba:0x0](ffff920a50c300c8) 00000080:00200000:2.0:1788572250.308795:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3bb:0x0](ffff920a50c2a1c8) 00000080:00200000:2.0:1788572250.308804:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3bb:0x0](ffff920a50c29148) 00000080:00200000:2.0:1788572250.308811:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3bc:0x0](ffff920a50c2ba88) 00000080:00200000:2.0:1788572250.308820:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3bc:0x0](ffff920a50c28908) 00000080:00200000:2.0:1788572250.308828:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3bd:0x0](ffff920a50c363c8) 00000080:00200000:2.0:1788572250.308838:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3bd:0x0](ffff920a50c34b08) 00000080:00200000:2.0:1788572250.308847:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3be:0x0](ffff920a50c31988) 00000080:00200000:2.0:1788572250.308857:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3be:0x0](ffff920a50c321c8) 00000080:00200000:2.0:1788572250.308865:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3bf:0x0](ffff920a50c33a88) 00000080:00200000:2.0:1788572250.308873:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3bf:0x0](ffff920a50c30908) 00000080:00200000:2.0:1788572250.308881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c0:0x0](ffff920a50c31148) 00000080:00200000:2.0:1788572250.308889:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c0:0x0](ffff920a50c33248) 00000080:00200000:2.0:1788572250.308897:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c1:0x0](ffff920a50c27448) 00000080:00200000:2.0:1788572250.308905:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c1:0x0](ffff920a50c200c8) 00000080:00200000:2.0:1788572250.308915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c2:0x0](ffff920a50c2c2c8) 00000080:00200000:2.0:1788572250.308924:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c2:0x0](ffff920a50c2f448) 00000080:00200000:2.0:1788572250.315753:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c3:0x0](ffff920a50c0ec08) 00000080:00200000:2.0:1788572250.315772:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c3:0x0](ffff920a50c0ba88) 00000080:00200000:2.0:1788572250.315783:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c4:0x0](ffff920a50c342c8) 00000080:00200000:2.0:1788572250.315795:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c4:0x0](ffff920a50c3db88) 00000080:00200000:2.0:1788572250.315805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c5:0x0](ffff920a50c263c8) 00000080:00200000:2.0:1788572250.315815:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c5:0x0](ffff920a50c24b08) 00000080:00200000:2.0:1788572250.315825:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c6:0x0](ffff920a50c2ec08) 00000080:00200000:2.0:1788572250.315835:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c6:0x0](ffff920a50c45b88) 00000080:00200000:2.0:1788572250.315846:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c7:0x0](ffff920a50c0cb08) 00000080:00200000:2.0:1788572250.315857:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c7:0x0](ffff920a50c0e3c8) 00000080:00200000:2.0:1788572250.315867:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c8:0x0](ffff920a50c0b248) 00000080:00200000:2.0:1788572250.315876:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c8:0x0](ffff920a50c09148) 00000080:00200000:2.0:1788572250.315887:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3c9:0x0](ffff920a50c22a08) 00000080:00200000:2.0:1788572250.315896:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3c9:0x0](ffff920a50c242c8) 00000080:00200000:2.0:1788572250.315906:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ca:0x0](ffff920a50c3ec08) 00000080:00200000:2.0:1788572250.315915:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ca:0x0](ffff920a50c3d348) 00000080:00200000:2.0:1788572250.315924:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3cb:0x0](ffff920a50c25b88) 00000080:00200000:2.0:1788572250.315946:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3cb:0x0](ffff920a50c20908) 00000080:00200000:2.0:1788572250.315957:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3cc:0x0](ffff920a50c47448) 00000080:00200000:2.0:1788572250.315968:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3cc:0x0](ffff920a50c400c8) 00000080:00200000:2.0:1788572250.315977:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3cd:0x0](ffff920a50c23a88) 00000080:00200000:2.0:1788572250.315986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3cd:0x0](ffff920a50c21988) 00000080:00200000:2.0:1788572250.315997:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ce:0x0](ffff920a50c44b08) 00000080:00200000:2.0:1788572250.316006:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ce:0x0](ffff920a50c41988) 00000080:00200000:2.0:1788572250.316015:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3cf:0x0](ffff920a50c4c2c8) 00000080:00200000:2.0:1788572250.316024:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3cf:0x0](ffff920a50c4aa08) 00000080:00200000:2.0:1788572250.316035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d0:0x0](ffff920a50c463c8) 00000080:00200000:2.0:1788572250.316044:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d0:0x0](ffff920a50c421c8) 00000080:00200000:2.0:1788572250.316080:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d1:0x0](ffff920a50c25348) 00000080:00200000:2.0:1788572250.316093:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d1:0x0](ffff920a50c23248) 00000080:00200000:2.0:1788572250.316103:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d2:0x0](ffff920a50c53a88) 00000080:00200000:2.0:1788572250.316124:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d2:0x0](ffff920a50c50908) 00000080:00200000:2.0:1788572250.316136:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d3:0x0](ffff920a50c4ec08) 00000080:00200000:2.0:1788572250.316146:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d3:0x0](ffff920a50c4d348) 00000080:00200000:2.0:1788572250.316156:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d4:0x0](ffff920a50c43a88) 00000080:00200000:2.0:1788572250.316166:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d4:0x0](ffff920a50c40908) 00000080:00200000:2.0:1788572250.316176:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d5:0x0](ffff920a50c52a08) 00000080:00200000:2.0:1788572250.316186:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d5:0x0](ffff920a50c55b88) 00000080:00200000:2.0:1788572250.316196:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d6:0x0](ffff920a50c3f448) 00000080:00200000:2.0:1788572250.316206:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d6:0x0](ffff920a50c380c8) 00000080:00200000:2.0:1788572250.316215:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d7:0x0](ffff920a50c542c8) 00000080:00200000:2.0:1788572250.316225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d7:0x0](ffff920a50c56c08) 00000080:00200000:2.0:1788572250.316235:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d8:0x0](ffff920a50c442c8) 00000080:00200000:2.0:1788572250.316247:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d8:0x0](ffff920a50c41148) 00000080:00200000:2.0:1788572250.316258:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3d9:0x0](ffff920a50c43248) 00000080:00200000:2.0:1788572250.316267:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3d9:0x0](ffff920a50c45348) 00000080:00200000:2.0:1788572250.316276:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3da:0x0](ffff920a50c46c08) 00000080:00200000:2.0:1788572250.316285:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3da:0x0](ffff920a50c42a08) 00000080:00200000:2.0:1788572250.316294:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3db:0x0](ffff920a50c4b248) 00000080:00200000:2.0:1788572250.316304:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3db:0x0](ffff920a50c4f448) 00000080:00200000:2.0:1788572250.316317:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3dc:0x0](ffff920a50c26c08) 00000080:00200000:2.0:1788572250.316326:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3dc:0x0](ffff920a50c5f448) 00000080:00200000:2.0:1788572250.316335:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3dd:0x0](ffff920a50c580c8) 00000080:00200000:2.0:1788572250.316343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3dd:0x0](ffff920a50c5e3c8) 00000080:00200000:2.0:1788572250.316352:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3de:0x0](ffff920a50c55348) 00000080:00200000:2.0:1788572250.316361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3de:0x0](ffff920a50c53248) 00000080:00200000:2.0:1788572250.316371:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3df:0x0](ffff920a50c5a1c8) 00000080:00200000:2.0:1788572250.316380:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3df:0x0](ffff920a50c59148) 00000080:00200000:2.0:1788572250.316390:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e0:0x0](ffff920a50c5ba88) 00000080:00200000:2.0:1788572250.316398:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e0:0x0](ffff920a50c58908) 00000080:00200000:2.0:1788572250.316407:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e1:0x0](ffff920a50c57448) 00000080:00200000:2.0:1788572250.316416:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e1:0x0](ffff920a50c500c8) 00000080:00200000:2.0:1788572250.316426:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e2:0x0](ffff920a50c5c2c8) 00000080:00200000:2.0:1788572250.316437:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e2:0x0](ffff920a50c5aa08) 00000080:00200000:2.0:1788572250.316446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e3:0x0](ffff920a50c5db88) 00000080:00200000:2.0:1788572250.316454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e3:0x0](ffff920a50c5ec08) 00000080:00200000:2.0:1788572250.316463:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e4:0x0](ffff920a50c5d348) 00000080:00200000:2.0:1788572250.316471:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e4:0x0](ffff920a50c5b248) 00000080:00200000:2.0:1788572250.316480:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e5:0x0](ffff920a50c51988) 00000080:00200000:2.0:1788572250.316490:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e5:0x0](ffff920a50c521c8) 00000080:00200000:2.0:1788572250.316499:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e6:0x0](ffff920a50c59988) 00000080:00200000:2.0:1788572250.316507:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e6:0x0](ffff920a50c5cb08) 00000080:00200000:2.0:1788572250.316515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e7:0x0](ffff920a50c63248) 00000080:00200000:2.0:1788572250.316524:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e7:0x0](ffff920a50c67448) 00000080:00200000:2.0:1788572250.316533:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e8:0x0](ffff920a50c4a1c8) 00000080:00200000:2.0:1788572250.316542:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e8:0x0](ffff920a50c49148) 00000080:00200000:2.0:1788572250.316551:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3e9:0x0](ffff920a50c663c8) 00000080:00200000:2.0:1788572250.316560:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3e9:0x0](ffff920a50c64b08) 00000080:00200000:2.0:1788572250.316571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ea:0x0](ffff920a50c39988) 00000080:00200000:2.0:1788572250.316748:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ea:0x0](ffff920a50c3a1c8) 00000080:00200000:2.0:1788572250.316763:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3eb:0x0](ffff920a50c51148) 00000080:00200000:2.0:1788572250.316773:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3eb:0x0](ffff920a50c54b08) 00000080:00200000:2.0:1788572250.316782:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ec:0x0](ffff920a50c4ba88) 00000080:00200000:2.0:1788572250.316791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ec:0x0](ffff920a50c48908) 00000080:00200000:2.0:1788572250.316800:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ed:0x0](ffff920a50c563c8) 00000080:00200000:2.0:1788572250.316810:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ed:0x0](ffff920a50c6d348) 00000080:00200000:2.0:1788572250.316819:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ee:0x0](ffff920a50c621c8) 00000080:00200000:2.0:1788572250.316827:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ee:0x0](ffff920a50c61148) 00000080:00200000:2.0:1788572250.316836:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ef:0x0](ffff920a50c6f448) 00000080:00200000:2.0:1788572250.316845:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ef:0x0](ffff920a50c680c8) 00000080:00200000:2.0:1788572250.316853:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f0:0x0](ffff920a50c6e3c8) 00000080:00200000:2.0:1788572250.316862:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f0:0x0](ffff920a50c6cb08) 00000080:00200000:2.0:1788572250.316871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f1:0x0](ffff920a50c60908) 00000080:00200000:2.0:1788572250.316881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f1:0x0](ffff920a50c642c8) 00000080:00200000:2.0:1788572250.316890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f2:0x0](ffff920a50c65b88) 00000080:00200000:2.0:1788572250.316901:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f2:0x0](ffff920a50c66c08) 00000080:00200000:2.0:1788572250.316910:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f3:0x0](ffff920a50c4db88) 00000080:00200000:2.0:1788572250.316919:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f3:0x0](ffff920a50c4e3c8) 00000080:00200000:2.0:1788572250.316938:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f4:0x0](ffff920a50c62a08) 00000080:00200000:2.0:1788572250.316949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f4:0x0](ffff920a50c63a88) 00000080:00200000:2.0:1788572250.316958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f5:0x0](ffff920a50c6a1c8) 00000080:00200000:2.0:1788572250.316967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f5:0x0](ffff920a50c69148) 00000080:00200000:2.0:1788572250.316976:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f6:0x0](ffff920a50c4cb08) 00000080:00200000:2.0:1788572250.316986:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f6:0x0](ffff920a50c49988) 00000080:00200000:2.0:1788572250.316994:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f7:0x0](ffff920a50c6ba88) 00000080:00200000:2.0:1788572250.317003:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f7:0x0](ffff920a50c68908) 00000080:00200000:2.0:1788572250.317012:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f8:0x0](ffff920a50c39148) 00000080:00200000:2.0:1788572250.317021:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f8:0x0](ffff920a50c3ba88) 00000080:00200000:2.0:1788572250.317031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3f9:0x0](ffff920a50c61988) 00000080:00200000:2.0:1788572250.317040:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3f9:0x0](ffff920a50c65348) 00000080:00200000:2.0:1788572250.317048:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3fa:0x0](ffff920a50c6c2c8) 00000080:00200000:2.0:1788572250.317057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3fa:0x0](ffff920a50c6aa08) 00000080:00200000:2.0:1788572250.317066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3fb:0x0](ffff920a50c6db88) 00000080:00200000:2.0:1788572250.317074:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3fb:0x0](ffff920a50c6ec08) 00000080:00200000:2.0:1788572250.317084:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3fc:0x0](ffff920a50c600c8) 00000080:00200000:2.0:1788572250.317093:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3fc:0x0](ffff920a50c76c08) 00000080:00200000:2.0:1788572250.317102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3fd:0x0](ffff920a50c480c8) 00000080:00200000:2.0:1788572250.317111:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3fd:0x0](ffff920a50c7a1c8) 00000080:00200000:2.0:1788572250.317290:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3fe:0x0](ffff920a50c75348) 00000080:00200000:2.0:1788572250.317300:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3fe:0x0](ffff920a50c73248) 00000080:00200000:2.0:1788572250.317310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x3ff:0x0](ffff920a50c6b248) 00000080:00200000:2.0:1788572250.317320:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x3ff:0x0](ffff920a50c69988) 00000080:00200000:2.0:1788572250.317329:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x400:0x0](ffff920a50c81148) 00000080:00200000:2.0:1788572250.317338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x400:0x0](ffff920a50c83a88) 00000080:00200000:2.0:1788572250.317348:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x401:0x0](ffff920a50c763c8) 00000080:00200000:2.0:1788572250.317358:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x401:0x0](ffff920a50c74b08) 00000080:00200000:2.0:1788572250.317367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x402:0x0](ffff920a50c7ba88) 00000080:00200000:2.0:1788572250.317376:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x402:0x0](ffff920a50c78908) 00000080:00200000:2.0:1788572250.317385:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x403:0x0](ffff920a50c3c2c8) 00000080:00200000:2.0:1788572250.317394:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x403:0x0](ffff920a50c3aa08) 00000080:00200000:2.0:1788572250.317417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x404:0x0](ffff920a50c7c2c8) 00000080:00200000:2.0:1788572250.317430:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x404:0x0](ffff920a50c7aa08) 00000080:00200000:2.0:1788572250.317439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x405:0x0](ffff920a50c842c8) 00000080:00200000:2.0:1788572250.317455:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x405:0x0](ffff920a50c82a08) 00000080:00200000:2.0:1788572250.317473:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x406:0x0](ffff920a50c80908) 00000080:00200000:2.0:1788572250.317488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x406:0x0](ffff920a50c85b88) 00000080:00200000:2.0:1788572250.317497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x407:0x0](ffff920a50c38908) 00000080:00200000:2.0:1788572250.317506:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x407:0x0](ffff920a50c3cb08) 00000080:00200000:2.0:1788572250.317515:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x408:0x0](ffff920a50c73a88) 00000080:00200000:2.0:1788572250.317525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x408:0x0](ffff920a50c70908) 00000080:00200000:2.0:1788572250.317534:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x409:0x0](ffff920a50c85348) 00000080:00200000:2.0:1788572250.317544:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x409:0x0](ffff920a50c83248) 00000080:00200000:2.0:1788572250.317553:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40a:0x0](ffff920a50c87448) 00000080:00200000:2.0:1788572250.317562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40a:0x0](ffff920a50c800c8) 00000080:00200000:2.0:1788572250.317571:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40b:0x0](ffff920a50c75b88) 00000080:00200000:2.0:1788572250.317581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40b:0x0](ffff920a50c72a08) 00000080:00200000:2.0:1788572250.317791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40c:0x0](ffff920a50c71148) 00000080:00200000:2.0:1788572250.317805:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40c:0x0](ffff920a50c721c8) 00000080:00200000:2.0:1788572250.317814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40d:0x0](ffff920a50c71988) 00000080:00200000:2.0:1788572250.317823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40d:0x0](ffff920a50c700c8) 00000080:00200000:2.0:1788572250.317833:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40e:0x0](ffff920a50c3b248) 00000080:00200000:2.0:1788572250.317842:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40e:0x0](ffff920a50c8cb08) 00000080:00200000:2.0:1788572250.317852:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x40f:0x0](ffff920a50c742c8) 00000080:00200000:2.0:1788572250.317861:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x40f:0x0](ffff920a50c97448) 00000080:00200000:2.0:1788572250.317871:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x410:0x0](ffff920a50c7ec08) 00000080:00200000:2.0:1788572250.317881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x410:0x0](ffff920a50c7d348) 00000080:00200000:2.0:1788572250.317891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x411:0x0](ffff920a50c863c8) 00000080:00200000:2.0:1788572250.317900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x411:0x0](ffff920a50c84b08) 00000080:00200000:2.0:1788572250.317908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x412:0x0](ffff920a50c89988) 00000080:00200000:2.0:1788572250.317918:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x412:0x0](ffff920a50c8a1c8) 00000080:00200000:2.0:1788572250.317938:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x413:0x0](ffff920a50c81988) 00000080:00200000:2.0:1788572250.317949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x413:0x0](ffff920a50c86c08) 00000080:00200000:2.0:1788572250.317958:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x414:0x0](ffff920a50c89148) 00000080:00200000:2.0:1788572250.317967:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x414:0x0](ffff920a50c8ba88) 00000080:00200000:2.0:1788572250.317977:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x415:0x0](ffff920a50c94b08) 00000080:00200000:2.0:1788572250.317987:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x415:0x0](ffff920a50c91988) 00000080:00200000:2.0:1788572250.317996:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x416:0x0](ffff920a50c921c8) 00000080:00200000:2.0:1788572250.318005:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x416:0x0](ffff920a50c91148) 00000080:00200000:2.0:1788572250.318014:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x417:0x0](ffff920a50c88908) 00000080:00200000:2.0:1788572250.318023:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x417:0x0](ffff920a50c8c2c8) 00000080:00200000:2.0:1788572250.318031:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x418:0x0](ffff920a50c8aa08) 00000080:00200000:2.0:1788572250.318039:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x418:0x0](ffff920a50c8db88) 00000080:00200000:2.0:1788572250.318049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x419:0x0](ffff920a50c90908) 00000080:00200000:2.0:1788572250.318058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x419:0x0](ffff920a50c942c8) 00000080:00200000:2.0:1788572250.318066:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41a:0x0](ffff920a50c7b248) 00000080:00200000:2.0:1788572250.318075:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41a:0x0](ffff920a50c7f448) 00000080:00200000:2.0:1788572250.318085:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41b:0x0](ffff920a50c821c8) 00000080:00200000:2.0:1788572250.318094:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41b:0x0](ffff920a50c3e3c8) 00000080:00200000:2.0:1788572250.318102:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41c:0x0](ffff920a50c8d348) 00000080:00200000:2.0:1788572250.318111:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41c:0x0](ffff920a50c8b248) 00000080:00200000:2.0:1788572250.318372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41d:0x0](ffff920a50c96c08) 00000080:00200000:2.0:1788572250.318399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41d:0x0](ffff920a50c95348) 00000080:00200000:2.0:1788572250.318411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41e:0x0](ffff920a50c93248) 00000080:00200000:2.0:1788572250.318420:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41e:0x0](ffff920a50c95b88) 00000080:00200000:2.0:1788572250.318428:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x41f:0x0](ffff920a50c7e3c8) 00000080:00200000:2.0:1788572250.318437:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x41f:0x0](ffff920a50c7cb08) 00000080:00200000:2.0:1788572250.318446:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x420:0x0](ffff920a50c9ba88) 00000080:00200000:2.0:1788572250.318456:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x420:0x0](ffff920a50c98908) 00000080:00200000:2.0:1788572250.318478:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x421:0x0](ffff920a50c9aa08) 00000080:00200000:2.0:1788572250.318488:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x421:0x0](ffff920a50c9db88) 00000080:00200000:2.0:1788572250.318496:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x422:0x0](ffff920a50c92a08) 00000080:00200000:2.0:1788572250.318505:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x422:0x0](ffff920a50c93a88) 00000080:00200000:2.0:1788572250.318513:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x423:0x0](ffff920a50c780c8) 00000080:00200000:2.0:1788572250.318522:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x423:0x0](ffff920a50c7db88) 00000080:00200000:2.0:1788572250.318531:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x424:0x0](ffff920a50c9ec08) 00000080:00200000:2.0:1788572250.318540:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x424:0x0](ffff920a50c9d348) 00000080:00200000:2.0:1788572250.318549:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x425:0x0](ffff920a50c963c8) 00000080:00200000:2.0:1788572250.318558:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x425:0x0](ffff920a50c900c8) 00000080:00200000:2.0:1788572250.318567:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x426:0x0](ffff920a50c880c8) 00000080:00200000:2.0:1788572250.318577:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x426:0x0](ffff920a50c8e3c8) 00000080:00200000:2.0:1788572250.319318:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x427:0x0](ffff920a50c9c2c8) 00000080:00200000:2.0:1788572250.319341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x427:0x0](ffff920a50c9b248) 00000080:00200000:2.0:1788572250.319351:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x428:0x0](ffff920a50c79988) 00000080:00200000:2.0:1788572250.319362:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x428:0x0](ffff920a50c77448) 00000080:00200000:2.0:1788572250.319372:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x429:0x0](ffff920a50ca63c8) 00000080:00200000:2.0:1788572250.319382:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x429:0x0](ffff920a50ca4b08) 00000080:00200000:2.0:1788572250.319391:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42a:0x0](ffff920a50cab248) 00000080:00200000:2.0:1788572250.319402:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42a:0x0](ffff920a50caf448) 00000080:00200000:2.0:1788572250.319411:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42b:0x0](ffff920a50ca21c8) 00000080:00200000:2.0:1788572250.319421:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42b:0x0](ffff920a50ca1148) 00000080:00200000:2.0:1788572250.319430:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42c:0x0](ffff920a50c9e3c8) 00000080:00200000:2.0:1788572250.319439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42c:0x0](ffff920a50c9cb08) 00000080:00200000:2.0:1788572250.319449:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42d:0x0](ffff920a50ca3a88) 00000080:00200000:2.0:1788572250.319978:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42d:0x0](ffff920a50ca0908) 00000080:00200000:2.0:1788572250.320000:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42e:0x0](ffff920a50c99988) 00000080:00200000:2.0:1788572250.320013:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42e:0x0](ffff920a50c980c8) 00000080:00200000:2.0:1788572250.320025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x42f:0x0](ffff920a50cad348) 00000080:00200000:2.0:1788572250.320037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x42f:0x0](ffff920a50ca80c8) 00000080:00200000:2.0:1788572250.320047:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x430:0x0](ffff920a50c99148) 00000080:00200000:2.0:1788572250.320057:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x430:0x0](ffff920a50c9a1c8) 00000080:00200000:2.0:1788572250.320067:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x431:0x0](ffff920a50ca2a08) 00000080:00200000:2.0:1788572250.320077:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x431:0x0](ffff920a50ca5b88) 00000080:00200000:2.0:1788572250.320086:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x432:0x0](ffff920a50cb2a08) 00000080:00200000:2.0:1788572250.320096:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x432:0x0](ffff920a50cb5b88) 00000080:00200000:2.0:1788572250.320106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x433:0x0](ffff920a50cacb08) 00000080:00200000:2.0:1788572250.320188:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x433:0x0](ffff920a50ca9988) 00000080:00200000:2.0:1788572250.320201:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x434:0x0](ffff920a50ca5348) 00000080:00200000:2.0:1788572250.320210:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x434:0x0](ffff920a50ca3248) 00000080:00200000:2.0:1788572250.320220:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x435:0x0](ffff920a50cb6c08) 00000080:00200000:2.0:1788572250.320229:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x435:0x0](ffff920a50cb5348) 00000080:00200000:2.0:1788572250.320239:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x436:0x0](ffff920a50cb3248) 00000080:00200000:2.0:1788572250.320249:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x436:0x0](ffff920a50cb7448) 00000080:00200000:2.0:1788572250.320281:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x437:0x0](ffff920a50c8ec08) 00000080:00200000:2.0:1788572250.320297:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x437:0x0](ffff920a50c8f448) 00000080:00200000:2.0:1788572250.320307:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x438:0x0](ffff920a50ca00c8) 00000080:00200000:2.0:1788572250.320316:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x438:0x0](ffff920a50ca7448) 00000080:00200000:2.0:1788572250.320326:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x439:0x0](ffff920a50c79148) 00000080:00200000:2.0:1788572250.320335:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x439:0x0](ffff920a50cba1c8) 00000080:00200000:2.0:1788572250.320345:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43a:0x0](ffff920a50cb63c8) 00000080:00200000:2.0:1788572250.320355:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43a:0x0](ffff920a50cb4b08) 00000080:00200000:2.0:1788572250.320364:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43b:0x0](ffff920a50cb9148) 00000080:00200000:2.0:1788572250.320373:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43b:0x0](ffff920a50cbba88) 00000080:00200000:2.0:1788572250.320383:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43c:0x0](ffff920a50ca6c08) 00000080:00200000:2.0:1788572250.320392:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43c:0x0](ffff920a50ca1988) 00000080:00200000:2.0:1788572250.320401:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43d:0x0](ffff920a50caa1c8) 00000080:00200000:2.0:1788572250.320412:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43d:0x0](ffff920a50ca9148) 00000080:00200000:2.0:1788572250.320421:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43e:0x0](ffff920a50caba88) 00000080:00200000:2.0:1788572250.320430:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43e:0x0](ffff920a50ca8908) 00000080:00200000:2.0:1788572250.320439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x43f:0x0](ffff920a50cb8908) 00000080:00200000:2.0:1788572250.320448:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x43f:0x0](ffff920a50cbc2c8) 00000080:00200000:2.0:1788572250.320513:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x440:0x0](ffff920a50ca42c8) 00000080:00200000:2.0:1788572250.320526:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x440:0x0](ffff920a50cc4b08) 00000080:00200000:2.0:1788572250.320536:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x441:0x0](ffff920a50cc21c8) 00000080:00200000:2.0:1788572250.320545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x441:0x0](ffff920a50cc1148) 00000080:00200000:2.0:1788572250.320554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x442:0x0](ffff920a50cac2c8) 00000080:00200000:2.0:1788572250.320564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x442:0x0](ffff920a50caaa08) 00000080:00200000:2.0:1788572250.320574:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x443:0x0](ffff920a50cb0908) 00000080:00200000:2.0:1788572250.320762:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x443:0x0](ffff920a50cb42c8) 00000080:00200000:2.0:1788572250.320778:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x444:0x0](ffff920a50cb3a88) 00000080:00200000:2.0:1788572250.320787:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x444:0x0](ffff920a50cb1148) 00000080:00200000:2.0:1788572250.320796:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x445:0x0](ffff920a50cb21c8) 00000080:00200000:2.0:1788572250.320806:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x445:0x0](ffff920a50cb1988) 00000080:00200000:2.0:1788572250.320814:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x446:0x0](ffff920a50cc0908) 00000080:00200000:2.0:1788572250.320823:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x446:0x0](ffff920a50cc42c8) 00000080:00200000:2.0:1788572250.320832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x447:0x0](ffff920a50cc2a08) 00000080:00200000:2.0:1788572250.320841:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x447:0x0](ffff920a50cc5b88) 00000080:00200000:2.0:1788572250.320850:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x448:0x0](ffff920a50cb00c8) 00000080:00200000:2.0:1788572250.320859:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x448:0x0](ffff920a50cce3c8) 00000080:00200000:2.0:1788572250.320870:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x449:0x0](ffff920a50cccb08) 00000080:00200000:2.0:1788572250.320878:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x449:0x0](ffff920a50cc9988) 00000080:00200000:2.0:1788572250.320888:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44a:0x0](ffff920a50cbec08) 00000080:00200000:2.0:1788572250.320898:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44a:0x0](ffff920a50cbd348) 00000080:00200000:2.0:1788572250.320912:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44b:0x0](ffff920a50cc6c08) 00000080:00200000:2.0:1788572250.320922:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44b:0x0](ffff920a50cc5348) 00000080:00200000:2.0:1788572250.320950:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44c:0x0](ffff920a50cbb248) 00000080:00200000:2.0:1788572250.320964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44c:0x0](ffff920a50cbf448) 00000080:00200000:2.0:1788572250.320973:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44d:0x0](ffff920a50ccba88) 00000080:00200000:2.0:1788572250.320982:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44d:0x0](ffff920a50cc8908) 00000080:00200000:2.0:1788572250.320992:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44e:0x0](ffff920a50cc00c8) 00000080:00200000:2.0:1788572250.321002:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44e:0x0](ffff920a50cc63c8) 00000080:00200000:2.0:1788572250.321010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x44f:0x0](ffff920a50ccaa08) 00000080:00200000:2.0:1788572250.321019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x44f:0x0](ffff920a50ccdb88) 00000080:00200000:2.0:1788572250.321027:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x450:0x0](ffff920a50cbe3c8) 00000080:00200000:2.0:1788572250.321053:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x450:0x0](ffff920a50cbcb08) 00000080:00200000:2.0:1788572250.321065:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x451:0x0](ffff920a50ccd348) 00000080:00200000:2.0:1788572250.321076:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x451:0x0](ffff920a50ccb248) 00000080:00200000:2.0:1788572250.321085:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x452:0x0](ffff920a50cc7448) 00000080:00200000:2.0:1788572250.321094:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x452:0x0](ffff920a50cc3248) 00000080:00200000:2.0:1788572250.321103:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x453:0x0](ffff920a50cb80c8) 00000080:00200000:2.0:1788572250.321114:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x453:0x0](ffff920a50cb9988) 00000080:00200000:2.0:1788572250.321242:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x454:0x0](ffff920a50ccf448) 00000080:00200000:2.0:1788572250.321255:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x454:0x0](ffff920a50cc80c8) 00000080:00200000:2.0:1788572250.321264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x455:0x0](ffff920a50cbaa08) 00000080:00200000:2.0:1788572250.321283:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x455:0x0](ffff920a50cbdb88) 00000080:00200000:2.0:1788572250.321292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x456:0x0](ffff920a50cadb88) 00000080:00200000:2.0:1788572250.321303:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x456:0x0](ffff920a50caec08) 00000080:00200000:2.0:1788572250.321313:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x457:0x0](ffff920a50cc3a88) 00000080:00200000:2.0:1788572250.321322:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x457:0x0](ffff920a50cd6c08) 00000080:00200000:2.0:1788572250.321333:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x458:0x0](ffff920a50ccc2c8) 00000080:00200000:2.0:1788572250.321342:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x458:0x0](ffff920a50cc9148) 00000080:00200000:2.0:1788572250.321357:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x459:0x0](ffff920a50cd5348) 00000080:00200000:2.0:1788572250.321367:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x459:0x0](ffff920a50cd3248) 00000080:00200000:2.0:1788572250.321377:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45a:0x0](ffff920a50cca1c8) 00000080:00200000:2.0:1788572250.321387:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45a:0x0](ffff920a50ccec08) 00000080:00200000:2.0:1788572250.321396:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45b:0x0](ffff920a50cd9988) 00000080:00200000:2.0:1788572250.321406:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45b:0x0](ffff920a50cda1c8) 00000080:00200000:2.0:1788572250.321415:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45c:0x0](ffff920a50cdba88) 00000080:00200000:2.0:1788572250.321424:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45c:0x0](ffff920a50cd8908) 00000080:00200000:2.0:1788572250.321433:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45d:0x0](ffff920a50cd63c8) 00000080:00200000:2.0:1788572250.321443:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45d:0x0](ffff920a50cd4b08) 00000080:00200000:2.0:1788572250.321454:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45e:0x0](ffff920a50cdc2c8) 00000080:00200000:2.0:1788572250.321477:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45e:0x0](ffff920a50cdaa08) 00000080:00200000:2.0:1788572250.321487:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x45f:0x0](ffff920a50cd1988) 00000080:00200000:2.0:1788572250.321497:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x45f:0x0](ffff920a50cd21c8) 00000080:00200000:2.0:1788572250.321507:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x460:0x0](ffff920a50ce2a08) 00000080:00200000:2.0:1788572250.321516:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x460:0x0](ffff920a50ce5b88) 00000080:00200000:2.0:1788572250.321525:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x461:0x0](ffff920a50cddb88) 00000080:00200000:2.0:1788572250.321535:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x461:0x0](ffff920a50cdec08) 00000080:00200000:2.0:1788572250.321545:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x462:0x0](ffff920a50ce6c08) 00000080:00200000:2.0:1788572250.321554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x462:0x0](ffff920a50ce5348) 00000080:00200000:2.0:1788572250.321562:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x463:0x0](ffff920a50ce7448) 00000080:00200000:2.0:1788572250.321572:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x463:0x0](ffff920a50ce00c8) 00000080:00200000:2.0:1788572250.321581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x464:0x0](ffff920a50cdd348) 00000080:00200000:2.0:1788572250.321758:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x464:0x0](ffff920a50cdb248) 00000080:00200000:2.0:1788572250.321771:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x465:0x0](ffff920a50cae3c8) 00000080:00200000:2.0:1788572250.321781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x465:0x0](ffff920a50c9f448) 00000080:00200000:2.0:1788572250.321790:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x466:0x0](ffff920a50cd3a88) 00000080:00200000:2.0:1788572250.321800:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x466:0x0](ffff920a50cd0908) 00000080:00200000:2.0:1788572250.321809:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x467:0x0](ffff920a50cde3c8) 00000080:00200000:2.0:1788572250.321817:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x467:0x0](ffff920a50cdcb08) 00000080:00200000:2.0:1788572250.321827:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x468:0x0](ffff920a50cd42c8) 00000080:00200000:2.0:1788572250.321836:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x468:0x0](ffff920a50cd2a08) 00000080:00200000:2.0:1788572250.321844:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x469:0x0](ffff920a50cd5b88) 00000080:00200000:2.0:1788572250.321852:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x469:0x0](ffff920a50cd1148) 00000080:00200000:2.0:1788572250.321878:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46a:0x0](ffff920a50cd00c8) 00000080:00200000:2.0:1788572250.321890:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46a:0x0](ffff920a50cd7448) 00000080:00200000:2.0:1788572250.321900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46b:0x0](ffff920a50cd80c8) 00000080:00200000:2.0:1788572250.321908:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46b:0x0](ffff920a50cdf448) 00000080:00200000:2.0:1788572250.321918:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46c:0x0](ffff920a50cd9148) 00000080:00200000:2.0:1788572250.321938:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46c:0x0](ffff920a50cf5b88) 00000080:00200000:2.0:1788572250.321949:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46d:0x0](ffff920a50ce8908) 00000080:00200000:2.0:1788572250.321964:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46d:0x0](ffff920a50cec2c8) 00000080:00200000:2.0:1788572250.321983:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46e:0x0](ffff920a50ceaa08) 00000080:00200000:2.0:1788572250.321992:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46e:0x0](ffff920a50cedb88) 00000080:00200000:2.0:1788572250.322001:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x46f:0x0](ffff920a50cc1988) 00000080:00200000:2.0:1788572250.322010:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x46f:0x0](ffff920a50cf8908) 00000080:00200000:2.0:1788572250.322019:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x470:0x0](ffff920a50cf3248) 00000080:00200000:2.0:1788572250.322028:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x470:0x0](ffff920a50cf7448) 00000080:00200000:2.0:1788572250.322037:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x471:0x0](ffff920a50cfc2c8) 00000080:00200000:2.0:1788572250.322045:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x471:0x0](ffff920a50cfaa08) 00000080:00200000:2.0:1788572250.322055:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x472:0x0](ffff920a50cf4b08) 00000080:00200000:2.0:1788572250.322064:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x472:0x0](ffff920a50cf1988) 00000080:00200000:2.0:1788572250.322074:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x473:0x0](ffff920a50cf21c8) 00000080:00200000:2.0:1788572250.322084:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x473:0x0](ffff920a50cf1148) 00000080:00200000:2.0:1788572250.322093:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x474:0x0](ffff920a50cfec08) 00000080:00200000:2.0:1788572250.322103:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x474:0x0](ffff920a50cfd348) 00000080:00200000:2.0:1788572250.322112:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x475:0x0](ffff920a50cf3a88) 00000080:00200000:2.0:1788572250.322204:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x475:0x0](ffff920a50cf0908) 00000080:00200000:2.0:1788572250.322214:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x476:0x0](ffff920a50cff448) 00000080:00200000:2.0:1788572250.322223:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x476:0x0](ffff920a50cf80c8) 00000080:00200000:2.0:1788572250.322232:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x477:0x0](ffff920a50cf42c8) 00000080:00200000:2.0:1788572250.322240:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x477:0x0](ffff920a50cf2a08) 00000080:00200000:2.0:1788572250.322249:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x478:0x0](ffff920a50cfb248) 00000080:00200000:2.0:1788572250.322259:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x478:0x0](ffff920a50cfe3c8) 00000080:00200000:2.0:1788572250.322269:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x479:0x0](ffff920a50cfcb08) 00000080:00200000:2.0:1788572250.322277:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x479:0x0](ffff920a50cf9988) 00000080:00200000:2.0:1788572250.322286:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x47a:0x0](ffff920a50ce21c8) 00000080:00200000:2.0:1788572250.322295:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x47a:0x0](ffff920a50ce1148) 00000080:00200000:2.0:1788572250.322304:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x2000255c0:0x47b:0x0](ffff920a50cf5348) 00000080:00200000:2.0:1788572250.322313:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240001789:0x47b:0x0](ffff920a50cf6c08) 00000080:00200000:2.0:1788572250.322323:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3:0x0](ffff920a50ce3a88) 00000080:00200000:2.0:1788572250.322332:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3:0x0](ffff920a50ce0908) 00000080:00200000:2.0:1788572250.322341:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x4:0x0](ffff920a50ce80c8) 00000080:00200000:2.0:1788572250.322350:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x4:0x0](ffff920a50cee3c8) 00000080:00200000:2.0:1788572250.322359:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x5:0x0](ffff920a50cf00c8) 00000080:00200000:2.0:1788572250.322369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x5:0x0](ffff920a50cf63c8) 00000080:00200000:2.0:1788572250.322379:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x6:0x0](ffff920a50d021c8) 00000080:00200000:2.0:1788572250.322388:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x6:0x0](ffff920a50d01148) 00000080:00200000:2.0:1788572250.322397:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x7:0x0](ffff920a50d03a88) 00000080:00200000:2.0:1788572250.322418:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x7:0x0](ffff920a50d00908) 00000080:00200000:2.0:1788572250.322429:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x8:0x0](ffff920a50ce9988) 00000080:00200000:2.0:1788572250.322439:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x8:0x0](ffff920a50cef448) 00000080:00200000:2.0:1788572250.322447:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x9:0x0](ffff920a50cf9148) 00000080:00200000:2.0:1788572250.322458:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x9:0x0](ffff920a50cfa1c8) 00000080:00200000:3.0:1788572250.323036:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xa:0x0](ffff920a50d02a08) 00000080:00200000:3.0:1788572250.323049:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xa:0x0](ffff920a50d05b88) 00000080:00200000:3.0:1788572250.323058:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xb:0x0](ffff920a50cfba88) 00000080:00200000:3.0:1788572250.323068:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xb:0x0](ffff920a50cfdb88) 00000080:00200000:3.0:1788572250.323077:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xc:0x0](ffff920a50d06c08) 00000080:00200000:3.0:1788572250.323087:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xc:0x0](ffff920a50d05348) 00000080:00200000:3.0:1788572250.323097:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xd:0x0](ffff920a50d03248) 00000080:00200000:3.0:1788572250.323106:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xd:0x0](ffff920a50d07448) 00000080:00200000:3.0:1788572250.323192:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xe:0x0](ffff920a50ce4b08) 00000080:00200000:3.0:1788572250.323205:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xe:0x0](ffff920a50ce63c8) 00000080:00200000:3.0:1788572250.323215:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0xf:0x0](ffff920a50d0f448) 00000080:00200000:3.0:1788572250.323225:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0xf:0x0](ffff920a50d080c8) 00000080:00200000:3.0:1788572250.323234:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x10:0x0](ffff920a50d0e3c8) 00000080:00200000:3.0:1788572250.323244:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x10:0x0](ffff920a50d0cb08) 00000080:00200000:3.0:1788572250.323254:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x11:0x0](ffff920a50d09988) 00000080:00200000:3.0:1788572250.323264:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x11:0x0](ffff920a50d0a1c8) 00000080:00200000:3.0:1788572250.323274:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x12:0x0](ffff920a50d000c8) 00000080:00200000:3.0:1788572250.323282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x12:0x0](ffff920a50d063c8) 00000080:00200000:3.0:1788572250.323292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x13:0x0](ffff920a50d09148) 00000080:00200000:3.0:1788572250.323301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x13:0x0](ffff920a50d0ba88) 00000080:00200000:3.0:1788572250.323310:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x14:0x0](ffff920a50d01988) 00000080:00200000:3.0:1788572250.323319:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x14:0x0](ffff920a50d04b08) 00000080:00200000:3.0:1788572250.323328:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x15:0x0](ffff920a50ce3248) 00000080:00200000:3.0:1788572250.323338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x15:0x0](ffff920a50ce42c8) 00000080:00200000:3.0:1788572250.323350:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x16:0x0](ffff920a50d042c8) 00000080:00200000:3.0:1788572250.323361:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x16:0x0](ffff920a50d121c8) 00000080:00200000:3.0:1788572250.323370:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x17:0x0](ffff920a50d0c2c8) 00000080:00200000:3.0:1788572250.323379:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x17:0x0](ffff920a50d0aa08) 00000080:00200000:3.0:1788572250.323388:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x18:0x0](ffff920a50d13a88) 00000080:00200000:3.0:1788572250.323399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x18:0x0](ffff920a50d10908) 00000080:00200000:3.0:1788572250.323408:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x19:0x0](ffff920a50ceb248) 00000080:00200000:3.0:1788572250.323417:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x19:0x0](ffff920a50ced348) 00000080:00200000:3.0:1788572250.323425:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1a:0x0](ffff920a50d0d348) 00000080:00200000:3.0:1788572250.323435:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1a:0x0](ffff920a50d0ec08) 00000080:00200000:3.0:1788572250.323443:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1b:0x0](ffff920a50d0db88) 00000080:00200000:3.0:1788572250.323452:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1b:0x0](ffff920a50d08908) 00000080:00200000:3.0:1788572250.323541:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1c:0x0](ffff920a50ce1988) 00000080:00200000:3.0:1788572250.323554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1c:0x0](ffff920a50d19988) 00000080:00200000:3.0:1788572250.323564:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1d:0x0](ffff920a50d12a08) 00000080:00200000:3.0:1788572250.323574:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1d:0x0](ffff920a50d15b88) 00000080:00200000:3.0:1788572250.323583:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1e:0x0](ffff920a50ceba88) 00000080:00200000:3.0:1788572250.323760:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1e:0x0](ffff920a50ce9148) 00000080:00200000:3.0:1788572250.323770:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x1f:0x0](ffff920a50d0b248) 00000080:00200000:3.0:1788572250.323781:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x1f:0x0](ffff920a50d242c8) 00000080:00200000:3.0:1788572250.323791:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x20:0x0](ffff920a50d22a08) 00000080:00200000:3.0:1788572250.323802:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x20:0x0](ffff920a50d25b88) 00000080:00200000:3.0:1788572250.323812:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x21:0x0](ffff920a50d26c08) 00000080:00200000:3.0:1788572250.323821:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x21:0x0](ffff920a50d25348) 00000080:00200000:3.0:1788572250.323832:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x22:0x0](ffff920a50d1ba88) 00000080:00200000:3.0:1788572250.323869:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x22:0x0](ffff920a50d18908) 00000080:00200000:3.0:1788572250.323881:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x23:0x0](ffff920a50d27448) 00000080:00200000:3.0:1788572250.323891:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x23:0x0](ffff920a50d200c8) 00000080:00200000:3.0:1788572250.323900:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x24:0x0](ffff920a50d1c2c8) 00000080:00200000:3.0:1788572250.323909:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x24:0x0](ffff920a50d1aa08) 00000080:00200000:3.0:1788572250.323917:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x25:0x0](ffff920a50d1db88) 00000080:00200000:3.0:1788572250.323938:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x25:0x0](ffff920a50d1ec08) 00000080:00200000:3.0:1788572250.323951:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x26:0x0](ffff920a50d221c8) 00000080:00200000:3.0:1788572250.323961:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x26:0x0](ffff920a50d21148) 00000080:00200000:3.0:1788572250.323970:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x27:0x0](ffff920a50cea1c8) 00000080:00200000:3.0:1788572250.323979:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x27:0x0](ffff920a50ceec08) 00000080:00200000:3.0:1788572250.323989:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x28:0x0](ffff920a50d23a88) 00000080:00200000:3.0:1788572250.324004:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x28:0x0](ffff920a50d20908) 00000080:00200000:3.0:1788572250.324015:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x29:0x0](ffff920a50d15348) 00000080:00200000:3.0:1788572250.324025:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x29:0x0](ffff920a50d13248) 00000080:00200000:3.0:1788572250.324035:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2a:0x0](ffff920a50d21988) 00000080:00200000:3.0:1788572250.324045:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2a:0x0](ffff920a50d24b08) 00000080:00200000:3.0:1788572250.324054:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2b:0x0](ffff920a50d29988) 00000080:00200000:3.0:1788572250.324064:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2b:0x0](ffff920a50d2a1c8) 00000080:00200000:3.0:1788572250.324073:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2c:0x0](ffff920a50d17448) 00000080:00200000:3.0:1788572250.324082:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2c:0x0](ffff920a50d100c8) 00000080:00200000:3.0:1788572250.324092:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2d:0x0](ffff920a50d1f448) 00000080:00200000:3.0:1788572250.324101:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2d:0x0](ffff920a50d180c8) 00000080:00200000:3.0:1788572250.324110:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2e:0x0](ffff920a50d14b08) 00000080:00200000:3.0:1788572250.324196:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2e:0x0](ffff920a50d11988) 00000080:00200000:3.0:1788572250.324210:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x2f:0x0](ffff920a50d28908) 00000080:00200000:3.0:1788572250.324220:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x2f:0x0](ffff920a50d2c2c8) 00000080:00200000:3.0:1788572250.324228:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x30:0x0](ffff920a50d2aa08) 00000080:00200000:3.0:1788572250.324238:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x30:0x0](ffff920a50d2db88) 00000080:00200000:3.0:1788572250.324247:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x31:0x0](ffff920a50d1cb08) 00000080:00200000:3.0:1788572250.324257:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x31:0x0](ffff920a50d1b248) 00000080:00200000:3.0:1788572250.324266:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x32:0x0](ffff920a50d1d348) 00000080:00200000:3.0:1788572250.324275:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x32:0x0](ffff920a50d19148) 00000080:00200000:3.0:1788572250.324282:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x33:0x0](ffff920a50d23248) 00000080:00200000:3.0:1788572250.324292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x33:0x0](ffff920a50d342c8) 00000080:00200000:3.0:1788572250.324301:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x34:0x0](ffff920a50d163c8) 00000080:00200000:3.0:1788572250.324311:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x34:0x0](ffff920a50d16c08) 00000080:00200000:3.0:1788572250.324320:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x35:0x0](ffff920a50d1e3c8) 00000080:00200000:3.0:1788572250.324330:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x35:0x0](ffff920a50d3d348) 00000080:00200000:3.0:1788572250.324338:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x36:0x0](ffff920a50d32a08) 00000080:00200000:3.0:1788572250.324347:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x36:0x0](ffff920a50d35b88) 00000080:00200000:3.0:1788572250.324356:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x37:0x0](ffff920a50d2d348) 00000080:00200000:3.0:1788572250.324365:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x37:0x0](ffff920a50d2b248) 00000080:00200000:3.0:1788572250.324374:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x38:0x0](ffff920a50d3f448) 00000080:00200000:3.0:1788572250.324384:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x38:0x0](ffff920a50d380c8) 00000080:00200000:3.0:1788572250.324393:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x39:0x0](ffff920a50d11148) 00000080:00200000:3.0:1788572250.324404:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x39:0x0](ffff920a50cecb08) 00000080:00200000:3.0:1788572250.324414:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3a:0x0](ffff920a50d3e3c8) 00000080:00200000:3.0:1788572250.324422:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3a:0x0](ffff920a50d3cb08) 00000080:00200000:3.0:1788572250.324431:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3b:0x0](ffff920a50d280c8) 00000080:00200000:3.0:1788572250.324441:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3b:0x0](ffff920a50d2e3c8) 00000080:00200000:3.0:1788572250.324540:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3c:0x0](ffff920a50d33248) 00000080:00200000:3.0:1788572250.324554:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3c:0x0](ffff920a50d37448) 00000080:00200000:3.0:1788572250.324563:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3d:0x0](ffff920a50d35348) 00000080:00200000:3.0:1788572250.324572:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3d:0x0](ffff920a50d300c8) 00000080:00200000:3.0:1788572250.324581:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3e:0x0](ffff920a50d363c8) 00000080:00200000:3.0:1788572250.327233:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3e:0x0](ffff920a50d34b08) 00000080:00200000:3.0:1788572250.327252:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x3f:0x0](ffff920a50d31988) 00000080:00200000:3.0:1788572250.327262:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x3f:0x0](ffff920a50d321c8) 00000080:00200000:3.0:1788572250.327273:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x40:0x0](ffff920a50d33a88) 00000080:00200000:3.0:1788572250.327283:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x40:0x0](ffff920a50d30908) 00000080:00200000:3.0:1788572250.327292:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x41:0x0](ffff920a50d44b08) 00000080:00200000:3.0:1788572250.327305:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x41:0x0](ffff920a50d41988) 00000080:00200000:3.0:1788572250.327315:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x42:0x0](ffff920a50d421c8) 00000080:00200000:3.0:1788572250.327325:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x42:0x0](ffff920a50d41148) 00000080:00200000:3.0:1788572250.327334:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x43:0x0](ffff920a50d40908) 00000080:00200000:3.0:1788572250.327343:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x43:0x0](ffff920a50d442c8) 00000080:00200000:3.0:1788572250.327351:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x44:0x0](ffff920a50d43a88) 00000080:00200000:3.0:1788572250.327360:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x44:0x0](ffff920a50d42a08) 00000080:00200000:3.0:1788572250.327369:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x45:0x0](ffff920a50d2cb08) 00000080:00200000:3.0:1788572250.327380:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x45:0x0](ffff920a50d2f448) 00000080:00200000:3.0:1788572250.327390:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x46:0x0](ffff920a50d31148) 00000080:00200000:3.0:1788572250.327399:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x46:0x0](ffff920a50d36c08) 00000080:00200000:3.0:1788572250.327408:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200025d90:0x47:0x0](ffff920a50d263c8) 00000080:00200000:3.0:1788572250.327418:0:160601:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002728:0x47:0x0](ffff920a50d4c2c8) 00000080:00200000:3.0:1788572250.327428:0:160601:0:(llite_lib.c:1688:ll_put_super()) VFS Op: cfg_instance lustre-client-ffff920a42f86800 (sb ffff920a42f86800) 00010000:00010000:3.0:1788572250.328236:0:160601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25bb26a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd626 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.328260:0:160601:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25bb26a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bd626 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.328268:0:160601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25bb26a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd626 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.328276:0:160601:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.157@tcp lock: ffff920a60a79800/0x8364c713e25bb26a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd626 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.330024:0:160601:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00010000:00010000:3.0:1788572250.330053:0:160601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a78400/0x8364c713e25bb263 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd61f expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.330064:0:160601:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a78400/0x8364c713e25bb263 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bd61f expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.330072:0:160601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a78400/0x8364c713e25bb263 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd61f expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.330079:0:160601:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.157@tcp lock: ffff920a60a78400/0x8364c713e25bb263 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd61f expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.334101:0:160601:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/0 locks to regular blp list, 0 blwis in pool 00010000:00010000:3.0:1788572250.334111:0:160601:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a78400/0x8364c713e25bb263 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd61f expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572250.334136:0:160601:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-cliir 00010000:00010000:3.0:1788572250.334148:0:160601:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.334156:0:160601:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.334164:0:160601:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.334170:0:160601:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.334190:0:160601:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 10000000:01000000:3.0:1788572250.334194:0:160601:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-client 10000000:01000000:3.0:1788572250.334197:0:160601:0:(mgc_request.c:547:config_log_end()) end config log lustre-client (0) 00020000:01000000:3.0:1788572250.334826:0:160601:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0000_UUID idx: 0 gen: 1 exp: ffff920a4596e800 active: 1 00020000:01000000:3.0:1788572250.334833:0:160601:0:(lov_obd.c:601:lov_del_target()) uuid: lustre-OST0001_UUID idx: 1 gen: 1 exp: ffff920a4596e000 active: 1 00020000:01000000:3.0:1788572250.334835:0:160601:0:(lov_obd.c:64:lov_tgts_putref()) destroying 2 lov targets 00020000:01000000:3.0:1788572250.334841:0:160601:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0001_UUID : lustre-OST0001-osc-ffff920a42f86800 00020000:01000000:3.0:1788572250.334843:0:160601:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff920a42f86800: disconnecting target lustre-OST0001-osc-ffff920a42f86800 00000100:00080000:3.0:1788572250.334856:0:160601:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-OST0001_UUID 00000100:00080000:3.0:1788572250.334998:0:160601:0:(import.c:65:import_set_state_nolock()) ffff920a45969800 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572250.335050:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569484288:192.168.202.157@tcp:9:umount.0 00010000:00010000:2.0:1788572250.335352:0:15201:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572250.335361:0:15201:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572250.335371:0:15201:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79600 already unused, calling callback (ffffffffc1336c20) 10000000:00010000:2.0:1788572250.335375:0:15201:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572250.335385:0:15201:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1008400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572250.335393:0:15201:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1009400000000 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572250.335399:0:15201:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 10000000:01000000:2.0:1788572250.335402:0:15201:0:(mgc_request.c:944:mgc_blocking_ast()) log params: stopping, won't requeue 10000000:01000000:2.0:1788572250.335405:0:15201:0:(mgc_request.c:130:config_log_put()) dropping config log params 00010000:00010000:2.0:1788572250.335431:0:15201:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 2/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572250.335444:0:15201:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572250.335470:0:15201:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 1/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572250.335479:0:15201:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79600/0x8364c713e25bb271 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd634 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572250.335494:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572250.335495:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572250.335497:0:15201:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572250.335504:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572250.335514:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569484416:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00080000:1.0:1788572250.350218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a45969800 lustre-OST0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572250.350286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569484288:192.168.202.157@tcp:9:umount.0 00000100:00080000:3.0:1788572250.351239:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1788572250.351248:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff920a42f86800: committing for last_committed 4294975670 gen 37 00000100:00100000:3.0:1788572250.351256:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.351481:0:160601:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x8364c713e2558408 00000020:00000080:3.0:1788572250.351497:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a4596e000/LOV_OSC_UUID 00000020:01000000:3.0:1788572250.351679:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0001-osc-ffff920a42f86800 (flags='') 00000020:00000080:3.0:1788572250.351686:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572250.351700:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-OST0001-osc-ffff920a42f86800: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788572250.351705:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 6 (ffff920a515219c0) has no exports 00000020:00000080:1.0:1788572250.353285:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a4596e000/LOV_OSC_UUID for lustre-OST0001-osc-ffff920a42f86800 00000008:01000000:3.0:1788572250.353494:0:160601:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0001-osc-ffff920a42f86800: client import never connected 00000100:00080000:3.0:1788572250.353501:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0001_UUID INVALID 00000100:00100000:3.0:1788572250.353507:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff920a42f86800: committing for last_committed 4294975670 gen 38 00000100:00100000:3.0:1788572250.353519:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.354425:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572250.354438:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0001-osc-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00020000:01000000:3.0:1788572250.354443:0:160601:0:(lov_obd.c:623:__lov_del_obd()) Removing tgt lustre-OST0000_UUID : lustre-OST0000-osc-ffff920a42f86800 00020000:01000000:3.0:1788572250.354445:0:160601:0:(lov_obd.c:248:lov_disconnect_obd()) lustre-clilov-ffff920a42f86800: disconnecting target lustre-OST0000-osc-ffff920a42f86800 00000100:00080000:3.0:1788572250.354471:0:160601:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d->lustre-OST0000_UUID 00000100:00080000:3.0:1788572250.354579:0:160601:0:(import.c:65:import_set_state_nolock()) ffff920a4596b800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000020:00000080:1.0:1788572250.354830:0:124398:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff920a45969800 for lustre-OST0001-osc-ffff920a42f86800 00000020:00000080:1.0:1788572250.354844:0:124398:0:(genops.c:885:class_export_put()) final put ffff920a45968800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:00000080:1.0:1788572250.354847:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a45968800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-OST0001-osc-ffff920a42f86800 00000020:01000000:1.0:1788572250.354854:0:124398:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0001-osc-ffff920a42f86800 (700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000100:00100000:2.0:1788572250.355268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569484544:192.168.202.157@tcp:9:umount.0 00000100:00100000:2.0:1788572250.357014:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:712ceec8-7c82-4602-bd77-3db1811a6b63:2400:1875444569484416:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00080000:1.0:1788572250.362658:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4596b800 lustre-OST0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572250.362714:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569484544:192.168.202.157@tcp:9:umount.0 00000100:00080000:3.0:1788572250.366638:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:3.0:1788572250.366648:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a42f86800: committing for last_committed 34359738472 gen 41 00000100:00100000:3.0:1788572250.366659:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.366821:0:160601:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x8364c713e2558401 00000020:00000080:3.0:1788572250.366844:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a4596e800/LOV_OSC_UUID 00000020:01000000:3.0:1788572250.366916:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-OST0000-osc-ffff920a42f86800 (flags='') 00000020:00000080:3.0:1788572250.366922:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572250.366942:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-OST0000-osc-ffff920a42f86800: forcing exports to disconnect: 0/3 00000020:00080000:3.0:1788572250.366947:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 5 (ffff920a51523380) has no exports 00000020:00000080:1.0:1788572250.367265:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a4596e800/LOV_OSC_UUID for lustre-OST0000-osc-ffff920a42f86800 00000008:01000000:3.0:1788572250.367346:0:160601:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-OST0000-osc-ffff920a42f86800: client import never connected 00000100:00080000:3.0:1788572250.367350:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-OST0000_UUID INVALID 00000100:00100000:3.0:1788572250.367353:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a42f86800: committing for last_committed 34359738472 gen 42 00000100:00100000:3.0:1788572250.367358:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.367689:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572250.367699:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-OST0000-osc-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:00000080:3.0:1788572250.367704:0:160601:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x8364c713e25583fa 00000020:00000080:3.0:1788572250.367711:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a4596a000/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:00000080:1.0:1788572250.373583:0:124398:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff920a4596b800 for lustre-OST0000-osc-ffff920a42f86800 00000020:00000080:1.0:1788572250.373615:0:124398:0:(genops.c:885:class_export_put()) final put ffff920a4596c000/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:00000080:1.0:1788572250.373619:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a4596c000/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-OST0000-osc-ffff920a42f86800 00000020:01000000:1.0:1788572250.373627:0:124398:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-OST0000-osc-ffff920a42f86800 (700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000100:00100000:1.0:1788572250.376863:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569485056:192.168.202.157@tcp:39:umount.0 00000100:00080000:1.0:1788572250.379237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413a000 lustre-MDT0000_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572250.379309:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569485056:192.168.202.157@tcp:39:umount.0 00000020:00000080:2.0:1788572250.383453:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a44b89000/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-MDT0000-mdc-ffff920a42f86800 00000100:00100000:1.0:1788572250.385353:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569485184:192.168.202.157@tcp:39:umount.0 00000100:00080000:1.0:1788572250.389753:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a5413b800 lustre-MDT0001_UUID: changing import state from CONNECTING to CLOSED 00000100:00100000:1.0:1788572250.389781:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:700c1fe3-c9bd-4b4d-876e-3774b25f2a3d:2396:1875444569485184:192.168.202.157@tcp:39:umount.0 00000020:00000080:2.0:1788572250.392772:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a44b8e000/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-MDT0001-mdc-ffff920a42f86800 00000020:00000080:2.0:1788572250.392809:0:160601:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x8364c713e25583e5 00000020:00000080:2.0:1788572250.392814:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a44b8e800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:01000000:2.0:1788572250.392833:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilov-ffff920a42f86800 (flags='') 00000020:00000080:2.0:1788572250.392842:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572250.392847:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-clilov-ffff920a42f86800: forcing exports to disconnect: 0/1 00000020:00080000:2.0:1788572250.392852:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 1 (ffff920a44394d40) has no exports 00000020:00000080:2.0:1788572250.392905:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572250.392912:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilov-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:00000080:2.0:1788572250.392916:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a44b8b800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:00000080:2.0:1788572250.392918:0:160601:0:(genops.c:844:class_export_destroy()) destroying export ffff920a44b8b800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-clilov-ffff920a42f86800 00000020:01000000:2.0:1788572250.392936:0:160601:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-clilov-ffff920a42f86800 (700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:01000000:2.0:1788572250.393121:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-clilmv-ffff920a42f86800 (flags='') 00000020:00000080:2.0:1788572250.393125:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572250.393129:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-clilmv-ffff920a42f86800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572250.393132:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 2 (ffff920a44393380) has no exports 00000020:00000080:2.0:1788572250.393169:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572250.393173:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-clilmv-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:01000000:2.0:1788572250.393177:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0000-mdc-ffff920a42f86800 (flags='') 00000020:00000080:2.0:1788572250.393180:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:2.0:1788572250.393182:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-MDT0000-mdc-ffff920a42f86800: forcing exports to disconnect: 0/2 00000020:00080000:2.0:1788572250.393184:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 3 (ffff920a54024d40) has no exports 00000008:01000000:2.0:1788572250.395966:0:160601:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0000-mdc-ffff920a42f86800: client import never connected 00000100:00080000:2.0:1788572250.395973:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0000_UUID INVALID 00000100:00100000:2.0:1788572250.395980:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a42f86800: committing for last_committed 339302416386 gen 13 00000100:00100000:2.0:1788572250.395989:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.404023:0:124398:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff920a5413a000 for lustre-MDT0000-mdc-ffff920a42f86800 00000020:00000080:2.0:1788572250.404233:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:2.0:1788572250.404244:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0000-mdc-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:00000080:2.0:1788572250.404250:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a5413d800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d 00000020:00000080:2.0:1788572250.404254:0:160601:0:(genops.c:844:class_export_destroy()) destroying export ffff920a5413d800/700c1fe3-c9bd-4b4d-876e-3774b25f2a3d for lustre-MDT0000-mdc-ffff920a42f86800 00000020:01000000:2.0:1788572250.404259:0:160601:0:(genops.c:441:class_free_dev()) finishing cleanup of obd lustre-MDT0000-mdc-ffff920a42f86800 (700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000020:01000000:3.0:1788572250.410488:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of lustre-MDT0001-mdc-ffff920a42f86800 (flags='') 00000020:00000080:3.0:1788572250.410498:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:3.0:1788572250.410506:0:160601:0:(obd_config.c:956:class_cleanup()) lustre-MDT0001-mdc-ffff920a42f86800: forcing exports to disconnect: 0/2 00000020:00080000:3.0:1788572250.410512:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 4 (ffff920a54023380) has no exports 00000008:01000000:3.0:1788572250.411205:0:160601:0:(obd_class.h:670:obd_cleanup_client_import()) lustre-MDT0001-mdc-ffff920a42f86800: client import never connected 00000100:00080000:3.0:1788572250.411211:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import lustre-MDT0001_UUID INVALID 00000100:00100000:3.0:1788572250.411215:0:160601:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a42f86800: committing for last_committed 17179869185 gen 3 00000100:00100000:3.0:1788572250.411222:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 00000020:00000080:3.0:1788572250.420371:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:3.0:1788572250.420388:0:160601:0:(obd_config.c:883:class_detach()) detach on obd lustre-MDT0001-mdc-ffff920a42f86800 (uuid 700c1fe3-c9bd-4b4d-876e-3774b25f2a3d) 00000080:02000400:3.0:1788572250.420395:0:160601:0:(llite_lib.c:1739:ll_put_super()) Unmounted lustre-client 00000020:01000000:3.0:1788572250.425292:0:160601:0:(obd_config.c:1198:class_del_profile()) Del profile lustre-client 00000020:01000004:3.0:1788572250.425352:0:160601:0:(obd_mount.c:897:lustre_common_put_super()) dropping sb ffff920a42f86800 00000100:00080000:3.0:1788572250.425361:0:160601:0:(pinger.c:398:ptlrpc_pinger_del_import()) removing pingable import 712ceec8-7c82-4602-bd77-3db1811a6b63->MGS 00010000:00010000:3.0:1788572250.425377:0:160601:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1008400000000 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572250.425388:0:160601:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x1009400000000 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572250.425395:0:160601:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 10000000:01000000:3.0:1788572250.425398:0:160601:0:(mgc_request.c:954:mgc_blocking_ast()) log lustre-s: disconnecting, won't requeue 10000000:01000000:3.0:1788572250.425401:0:160601:0:(mgc_request.c:130:config_log_put()) dropping config log lustre-sptlrpc 00010000:00010000:3.0:1788572250.425921:0:160601:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572250.426505:0:160601:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000100:00100000:3.0:1788572250.426526:0:160601:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a4805e300 going to sleep for 0 seconds 00000100:00100000:3.0:1788572250.426536:0:160601:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job umount:712ceec8-7c82-4602-bd77-3db1811a6b63:160601:1875444569485312:192.168.202.157@tcp:103:umount.0 00010000:00010000:3.0:1788572250.426640:0:160601:0:(ldlm_request.c:1625:ldlm_cli_cancel_req()) Got rc -108 from cancel RPC: canceling anyway 00010000:00010000:3.0:1788572250.426651:0:160601:0:(ldlm_request.c:2491:ldlm_cli_cancel_list()) ldlm_cli_cancel_list: -108 00010000:00010000:3.0:1788572250.426656:0:160601:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb278 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x5809400000020 nid: local remote: 0x265cfe11109bd642 expref: -99 pid: 15203 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572250.426690:0:160601:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from FULL to CONNECTING 00000100:00100000:3.0:1788572250.426721:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569485440:192.168.202.157@tcp:251:umount.0 00000100:00080000:3.0:1788572250.431502:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4596a800 MGS: changing import state from CONNECTING to CLOSED 10000000:01000000:3.0:1788572250.431511:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572250.431514:0:2396:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572250.431516:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:3.0:1788572250.431548:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:712ceec8-7c82-4602-bd77-3db1811a6b63:2396:1875444569485440:192.168.202.157@tcp:251:umount.0 00000100:00080000:1.0:1788572250.434556:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572250.434568:0:160601:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572250.434573:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572250.434576:0:160601:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:1.0:1788572250.434604:0:160601:0:(genops.c:1468:class_disconnect()) disconnect: cookie 0x8364c713e2558398 00000020:00000080:1.0:1788572250.434616:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a44b8d000/712ceec8-7c82-4602-bd77-3db1811a6b63 00000020:01000000:1.0:1788572250.434640:0:160601:0:(obd_config.c:2477:class_manual_cleanup()) Manual cleanup of MGC192.168.202.157@tcp (flags='') 00000020:00000080:1.0:1788572250.434650:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf004 00000020:00000080:1.0:1788572250.434657:0:160601:0:(obd_config.c:956:class_cleanup()) MGC192.168.202.157@tcp: forcing exports to disconnect: 0/3 00000020:00080000:1.0:1788572250.434662:0:160601:0:(genops.c:1558:class_disconnect_exports()) OBD device 0 (ffff920a51524d40) has no exports 10000000:01000000:1.0:1788572250.434704:0:15203:0:(mgc_request.c:708:mgc_requeue_thread()) Ending requeue thread 00000020:00000080:3.0:1788572250.434735:0:124398:0:(genops.c:844:class_export_destroy()) destroying export ffff920a44b8d000/712ceec8-7c82-4602-bd77-3db1811a6b63 for MGC192.168.202.157@tcp 10000000:01000000:1.0:1788572250.434867:0:160601:0:(obd_class.h:670:obd_cleanup_client_import()) MGC192.168.202.157@tcp: client import never connected 00000100:00080000:1.0:1788572250.434871:0:160601:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 10000000:01000000:1.0:1788572250.434873:0:160601:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00100000:1.0:1788572250.434875:0:160601:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:1.0:1788572250.434877:0:160601:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00000020:00000080:3.0:1788572250.434939:0:124398:0:(genops.c:1183:obd_zombie_import_free()) destroying import ffff920a4596a800 for MGC192.168.202.157@tcp 00000020:00000080:1.0:1788572250.434968:0:160601:0:(obd_config.c:1472:class_process_config()) processing cmd: cf002 00000020:00000080:1.0:1788572250.434974:0:160601:0:(obd_config.c:883:class_detach()) detach on obd MGC192.168.202.157@tcp (uuid 712ceec8-7c82-4602-bd77-3db1811a6b63) 00000020:00000080:1.0:1788572250.434977:0:160601:0:(genops.c:885:class_export_put()) final put ffff920a42f82000/712ceec8-7c82-4602-bd77-3db1811a6b63 00000020:00000080:1.0:1788572250.434979:0:160601:0:(genops.c:844:class_export_destroy()) destroying export ffff920a42f82000/712ceec8-7c82-4602-bd77-3db1811a6b63 for MGC192.168.202.157@tcp 00000020:01000000:1.0:1788572250.434984:0:160601:0:(genops.c:441:class_free_dev()) finishing cleanup of obd MGC192.168.202.157@tcp (712ceec8-7c82-4602-bd77-3db1811a6b63) 00010000:00010000:3.0:1788572250.435513:0:15200:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-1 blwis in pool) 00010000:00010000:3.0:1788572250.436018:0:33368:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-2 blwis in pool) 00010000:00010000:2.0:1788572250.436132:0:15201:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-3 blwis in pool) 00010000:00010000:3.0:1788572250.436356:0:33367:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (-4 blwis in pool) 00000100:00100000:1.0:1788572250.436663:0:15190:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_000: service thread [ffff920a45b80a00:15190] 0 exiting: rc = 1 00000100:00100000:1.0:1788572250.436727:0:116376:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_002: service thread [ffff920a5419ae00:116376] 2 exiting: rc = 0 00000100:00100000:1.0:1788572250.438738:0:15191:0:(service.c:3183:ptlrpc_main()) ldlm_cb00_001: service thread [ffff920a45b80c00:15191] 1 exiting: rc = 0 00000100:00100000:3.0:1788572250.439095:0:15193:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_001: service thread [ffff920a45b81e00:15193] 1 exiting: rc = 0 00000100:00100000:2.0:1788572250.439118:0:15192:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_000: service thread [ffff920a45b81c00:15192] 0 exiting: rc = 1 00000100:00100000:3.0:1788572250.439247:0:33366:0:(service.c:3183:ptlrpc_main()) ldlm_cb01_002: service thread [ffff920a525cb600:33366] 2 exiting: rc = 0 00000100:00100000:3.0:1788572250.439627:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:3.0:1788572250.439638:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.439986:0:15194:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_000: service thread [ffff920a45b81200:15194] 0 exiting: rc = 1 00000100:00100000:1.0:1788572250.440050:0:15195:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_001: service thread [ffff920a4422ee00:15195] 1 exiting: rc = 0 00000100:00100000:1.0:1788572250.440100:0:15196:0:(service.c:3183:ptlrpc_main()) ldlm_cn00_002: service thread [ffff920a4422f800:15196] 2 exiting: rc = 0 00000100:00100000:3.0:1788572250.440653:0:15198:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_001: service thread [ffff920a60827000:15198] 1 exiting: rc = 0 00000100:00100000:3.0:1788572250.440705:0:15197:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_000: service thread [ffff920a4422f600:15197] 0 exiting: rc = 1 00000100:00100000:2.0:1788572250.440707:0:15199:0:(service.c:3183:ptlrpc_main()) ldlm_cn01_002: service thread [ffff920a54310e00:15199] 2 exiting: rc = 0 00000100:00100000:1.0:1788572250.440916:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440923:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440939:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440945:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440951:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440955:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440961:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440966:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440972:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440977:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440983:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440988:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440994:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.440999:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441005:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441010:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441016:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441023:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441028:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441034:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441040:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441046:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441051:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441057:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441063:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441069:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441074:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441079:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441085:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441090:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441097:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441101:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441107:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441112:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441118:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441123:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441130:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441135:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441140:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441145:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441151:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441159:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441164:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441170:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441176:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441183:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441189:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441196:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441202:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441208:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441215:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441222:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441228:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441234:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441240:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441245:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441251:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441257:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441263:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441269:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441275:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441280:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441285:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441290:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441298:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441305:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441310:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441317:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441323:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441329:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441334:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441340:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441346:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441352:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441357:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441363:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441369:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441387:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441393:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441400:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441406:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441412:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441417:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441424:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441429:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441435:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441440:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441445:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441451:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441456:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441472:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441478:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441484:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441490:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441495:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441501:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441507:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441511:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441516:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441521:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441527:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441532:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441537:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441542:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441547:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441552:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441558:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441563:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441568:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441573:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441579:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441607:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441613:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441619:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441624:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441629:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441634:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441639:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441644:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441649:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441655:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441661:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441666:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441670:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441675:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441681:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441686:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000100:00100000:1.0:1788572250.441691:0:160601:0:(events.c:344:request_in_callback()) peer: 0-0@<0:0> (source: 0-0@<0:0>) 00000020:01000004:1.0:1788572250.443120:0:160601:0:(obd_mount.c:710:lustre_put_lsi()) put ffff920a42f86800 1 00000020:01000004:1.0:1788572250.443133:0:160601:0:(obd_mount.c:668:lustre_free_lsi()) Freeing lsi ffff920a42f86000 00000020:01000004:3.0:1788572299.657844:0:160918:0:(obd_mount.c:923:lmd_print()) mount data: 00000020:01000004:3.0:1788572299.657849:0:160918:0:(obd_mount.c:925:lmd_print()) profile: lustre-client 00000020:01000004:3.0:1788572299.657850:0:160918:0:(obd_mount.c:926:lmd_print()) device: 192.168.202.157@tcp:/lustre 00000020:01000004:3.0:1788572299.657851:0:160918:0:(obd_mount.c:929:lmd_print()) options: user_xattr,flock 00000080:01200004:3.0:1788572299.657955:0:160918:0:(super25.c:131:lustre_fill_super()) VFS Op: sb ffff920a47b93800 00000080:01000004:3.0:1788572299.657964:0:160918:0:(super25.c:162:lustre_fill_super()) Mounting client lustre-client 00000020:01000004:3.0:1788572299.657980:0:160918:0:(obd_mount.c:411:lustre_start_mgc()) Start MGC 'MGC192.168.202.157@tcp' 00000020:01000004:3.0:1788572299.658007:0:160918:0:(obd_mount.c:230:lustre_start_simple()) Starting OBD MGC192.168.202.157@tcp (typ=mgc) 00000020:00000080:3.0:1788572299.658019:0:160918:0:(genops.c:413:class_newdev()) Allocate new device MGC192.168.202.157@tcp (ffff920a44393380) 00000020:00000080:3.0:1788572299.658031:0:160918:0:(obd_config.c:720:class_attach_name()) OBD: dev 0 attached type mgc with refcount 2 00000100:00100000:3.0:1788572299.658198:0:160918:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:3.0:1788572299.658223:0:160918:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/1 left), rc = 0 00000100:00100000:3.0:1788572299.658360:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[0] started 0 min 2 max 34 00000100:00100000:3.0:1788572299.658366:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb00_000' 00000100:00100000:3.0:1788572299.660004:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 0 min 2 max 34 00000100:00100000:3.0:1788572299.660011:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_000' 00000100:00100000:3.0:1788572299.660748:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_cbd[1] started 1 min 2 max 34 00000100:00100000:3.0:1788572299.660757:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cb01_001' 00000100:00100000:3.0:1788572299.665430:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[0] started 2 min 3 max 34 00000100:00100000:3.0:1788572299.665440:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn00_002' 00000100:00100000:3.0:1788572299.666092:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 0 min 3 max 34 00000100:00100000:3.0:1788572299.666101:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_000' 00000100:00100000:3.0:1788572299.666963:0:160918:0:(service.c:3462:ptlrpc_start_thread()) ldlm_canceld[1] started 1 min 3 max 34 00000100:00100000:3.0:1788572299.666972:0:160918:0:(service.c:3521:ptlrpc_start_thread()) starting thread 'ldlm_cn01_001' 00010000:00010000:3.0:1788572299.668765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572299.668768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572299.668770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:3.0:1788572299.669550:0:160918:0:(ldlm_lib.c:102:import_set_conn()) imp ffff920a54164800@MGC192.168.202.157@tcp: add connection 192.168.202.157@tcp at head 00010000:00010000:3.0:1788572299.669745:0:160918:0:(ldlm_pool.c:923:ldlm_pool_init()) Lock pool ldlm-pool-MGC192.168.202.157@tcp-0 is initialized 00000040:01000000:3.0:1788572299.669752:0:160918:0:(llog_obd.c:197:llog_setup()) obd MGC192.168.202.157@tcp ctxt 1 is initialized 10000000:01000000:1.0:1788572299.672669:0:160932:0:(mgc_request.c:631:mgc_requeue_thread()) Starting requeue thread 00000020:00000080:3.0:1788572299.672764:0:160918:0:(obd_config.c:826:class_setup()) finished setup of obd MGC192.168.202.157@tcp (uuid f79285ed-4685-45a8-8731-5e2522a143f3) 00000020:00000080:3.0:1788572299.672790:0:160918:0:(genops.c:1385:class_connect()) connect: client f79285ed-4685-45a8-8731-5e2522a143f3, cookie 0x8364c713e25bb2a2 00000100:00080000:3.0:1788572299.672804:0:160918:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:3.0:1788572299.672816:0:160918:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572299.672849:0:160918:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3df80 x1875444569485568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572299.672897:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569485568:192.168.202.157@tcp:250:mount.lustre.0 00000100:00080000:3.0:1788572299.673371:0:160918:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import f79285ed-4685-45a8-8731-5e2522a143f3->MGS 00000080:00200000:3.0:1788572299.673402:0:160918:0:(llite_lib.c:1506:ll_fill_super()) VFS Op: cfg_instance lustre-client-ffff920a47b93800 (sb ffff920a47b93800) 00000080:01000000:3.0:1788572299.673476:0:160918:0:(llite_lib.c:1232:ll_options()) Parsing opts user_xattr,flock 00000080:00000004:3.0:1788572299.673478:0:160918:0:(llite_lib.c:1242:ll_options()) next opt=user_xattr 00000080:00000004:3.0:1788572299.673482:0:160918:0:(llite_lib.c:1242:ll_options()) next opt=flock 00000080:01000000:3.0:1788572299.673489:0:160918:0:(llite_lib.c:1540:ll_fill_super()) llite sb uuid: 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b 10000000:01000000:3.0:1788572299.674914:0:160918:0:(mgc_request.c:1944:mgc_process_config()) parse_log lustre-client from 0 10000000:01000000:3.0:1788572299.674919:0:160918:0:(mgc_request.c:329:config_log_add()) add config log lustre-client-ffff920a47b93800 10000000:01000000:3.0:1788572299.674923:0:160918:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-sptlrpc-ffff920a44393380 10000000:01000000:3.0:1788572299.674927:0:160918:0:(mgc_request.c:68:mgc_name2resid()) log lustre-sptlrpc to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572299.674934:0:160918:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff920a44393380 from 1 10000000:01000000:3.0:1788572299.674939:0:160918:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:3.0:1788572299.674981:0:160918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.674992:0:160918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.675003:0:160918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572299.676136:0:160918:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a51b3f480 x1875444569485696/t0(0) o101->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572299.676155:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 0 seconds 00010000:00010000:3.0:1788572299.676983:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572299.678122:0:2396:0:(client.c:3126:ptlrpc_free_committed()) MGC192.168.202.157@tcp: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572299.678138:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.157@tcp: connect to target with instance 0 00000100:00080000:3.0:1788572299.678148:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) MGC192.168.202.157@tcp: Resetting ns_connect_flags to server flags: 0xa000011001002060 10000000:01000000:3.0:1788572299.678154:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:00080000:3.0:1788572299.678159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to FULL 10000000:01000000:3.0:1788572299.678162:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572299.683705:0:2396:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:00080000:3.0:1788572299.683726:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000008fde49a9): req@ffff920a51b3f480 x1875444569485696/t0(0) o101->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 328/344 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572299.683757:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569485568:192.168.202.157@tcp:250:mount.lustre.0 00000100:00100000:3.0:1788572299.684415:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.691498:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569485696:192.168.202.157@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572299.691540:0:160918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.691553:0:160918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.691644:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569485824:192.168.202.157@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572299.691817:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.695903:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569485824:192.168.202.157@tcp:501:mount.lustre.0 10000000:01000000:3.0:1788572299.695952:0:160918:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:3.0:1788572299.695973:0:160918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.695985:0:160918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572299.695994:0:160918:0:(mgc_request.c:194:do_config_log_add()) do adding config log params-ffff920a47b93800 10000000:01000000:3.0:1788572299.695999:0:160918:0:(mgc_request.c:68:mgc_name2resid()) log params to resid 0x736d61726170/0x3 (params) 10000000:01000000:3.0:1788572299.696004:0:160918:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-client-ffff920a47b93800 10000000:01000000:3.0:1788572299.696006:0:160918:0:(mgc_request.c:68:mgc_name2resid()) log lustre-client to resid 0x65727473756c/0x0 (lustre) 10000000:01000000:3.0:1788572299.696010:0:160918:0:(mgc_request.c:194:do_config_log_add()) do adding config log lustre-cliir-ffff920a47b93800 10000000:01000000:3.0:1788572299.696012:0:160918:0:(mgc_request.c:68:mgc_name2resid()) log lustre-cliir to resid 0x65727473756c/0x2 (lustre) 10000000:01000000:3.0:1788572299.696018:0:160918:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff920a47b93800 from 1 10000000:01000000:3.0:1788572299.696021:0:160918:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:3.0:1788572299.696066:0:160918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.696075:0:160918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.696085:0:160918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.696106:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569485952:192.168.202.157@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572299.696261:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.719763:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569485952:192.168.202.157@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572299.719794:0:160918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.719809:0:160918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.719895:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486080:192.168.202.157@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572299.719963:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572299.785443:0:160937:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160937:1875444569486464:192.168.202.157@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572299.785489:0:160937:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:1.0:1788572299.785500:0:160937:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572299.785502:0:160937:0:(obd_config.c:1563:class_process_config()) marker 40 (0x1) lustre sys.jobid_var 00000020:00000080:1.0:1788572299.785509:0:160937:0:(obd_config.c:1472:class_process_config()) processing cmd: cf00f 00000100:00100000:1.0:1788572299.785546:0:160937:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160937:1875444569486592:192.168.202.157@tcp:502:llog_process_th.0 00000100:00100000:1.0:1788572299.785651:0:160937:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af50c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572299.797966:0:160937:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160937:1875444569486592:192.168.202.157@tcp:502:llog_process_th.0 00000020:01000000:1.0:1788572299.798013:0:160937:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:1.0:1788572299.798022:0:160937:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:1.0:1788572299.798025:0:160937:0:(obd_config.c:1563:class_process_config()) marker 40 (0x2) lustre sys.jobid_var 00000040:00080000:1.0:1788572299.798030:0:160937:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 52 count 51 00000020:01000000:3.0:1788572299.798425:0:160918:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 1-51 (rc=0) 10000000:01000000:3.0:1788572299.798445:0:160918:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:3.0:1788572299.798458:0:160918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.798469:0:160918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572299.798479:0:160918:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 1 10000000:01000000:3.0:1788572299.798482:0:160918:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:3.0:1788572299.798519:0:160918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.798527:0:160918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.798538:0:160918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.798559:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486720:192.168.202.157@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572299.798657:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.802796:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486720:192.168.202.157@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572299.802842:0:160918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc4d expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.802857:0:160918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdc4d expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.803453:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486848:192.168.202.157@tcp:256:mount.lustre.0 00000100:00100000:3.0:1788572299.803552:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.809827:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486848:192.168.202.157@tcp:256:mount.lustre.0 10000000:01000000:3.0:1788572299.810539:0:160918:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:3.0:1788572299.810661:0:160918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc4d expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.810686:0:160918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc4d expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572299.810696:0:160918:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff920a47b93800 from 1 10000000:01000000:3.0:1788572299.810698:0:160918:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:3.0:1788572299.810743:0:160918:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.810751:0:160918:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.810760:0:160918:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.810783:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486976:192.168.202.157@tcp:101:mount.lustre.0 00000100:00100000:3.0:1788572299.810861:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.816527:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569486976:192.168.202.157@tcp:101:mount.lustre.0 00010000:00010000:3.0:1788572299.816558:0:160918:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572299.816573:0:160918:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572299.816709:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569487104:192.168.202.157@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572299.816782:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.825388:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569487104:192.168.202.157@tcp:501:mount.lustre.0 00000100:00100000:3.0:1788572299.825465:0:160918:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569487232:192.168.202.157@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572299.825566:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.834827:0:160918:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job mount.lustre:f79285ed-4685-45a8-8731-5e2522a143f3:160918:1875444569487232:192.168.202.157@tcp:503:mount.lustre.0 00000100:00100000:3.0:1788572299.840007:0:160944:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160944:1875444569487360:192.168.202.157@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572299.840120:0:160944:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.842649:0:160944:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160944:1875444569487360:192.168.202.157@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572299.842710:0:160944:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572299.842721:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572299.842723:0:160944:0:(obd_config.c:1563:class_process_config()) marker 2 (0x1) general debug_raw_pointers 00000020:01000000:3.0:1788572299.842731:0:160944:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff920a47b93800 00000020:00000080:3.0:1788572299.842734:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000020:01000000:3.0:1788572299.842742:0:160944:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572299.842744:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572299.842746:0:160944:0:(obd_config.c:1563:class_process_config()) marker 2 (0x2) general debug_raw_pointers 00000020:01000000:3.0:1788572299.842749:0:160944:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x0 mark_flg=0x1 00000020:00000080:3.0:1788572299.842751:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572299.842752:0:160944:0:(obd_config.c:1563:class_process_config()) marker 3 (0x1) general lod.*.mdt_hash 00000020:01000000:3.0:1788572299.842756:0:160944:0:(obd_config.c:2004:class_config_llog_handler()) cmd ce032, instance name: general-ffff920a47b93800 00000020:00000080:3.0:1788572299.842758:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: ce032 00000100:00100000:3.0:1788572299.842852:0:160944:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160944:1875444569487488:192.168.202.157@tcp:502:llog_process_th.0 00000100:00100000:3.0:1788572299.842948:0:160944:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572299.846955:0:160944:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job llog_process_th:f79285ed-4685-45a8-8731-5e2522a143f3:160944:1875444569487488:192.168.202.157@tcp:502:llog_process_th.0 00000020:01000000:3.0:1788572299.847003:0:160944:0:(obd_config.c:1893:class_config_llog_handler()) Marker, inst_flg=0x2 mark_flg=0x2 00000020:00000080:3.0:1788572299.847011:0:160944:0:(obd_config.c:1472:class_process_config()) processing cmd: cf010 00000020:00000080:3.0:1788572299.847014:0:160944:0:(obd_config.c:1563:class_process_config()) marker 3 (0x2) general lod.*.mdt_hash 00000040:00080000:3.0:1788572299.847020:0:160944:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 7 count 7 00000020:01000000:1.0:1788572299.857975:0:160918:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 1-6 (rc=0) 10000000:01000000:1.0:1788572299.858014:0:160918:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:1.0:1788572299.858041:0:160918:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572299.858055:0:160918:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:01000000:1.0:1788572299.858077:0:160918:0:(llite_lib.c:1614:ll_fill_super()) Found profile lustre-client: mdc=lustre-clilmv osc=lustre-clilov 00000020:00000080:1.0:1788572299.858103:0:160918:0:(genops.c:1385:class_connect()) connect: client 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b, cookie 0x8364c713e25bb2ef 00800000:01000000:1.0:1788572299.858127:0:160918:0:(lmv_obd.c:515:lmv_check_connect()) Time to connect 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b to lustre-clilmv-ffff920a47b93800 00800000:01000000:1.0:1788572299.858134:0:160918:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0000-mdc-ffff920a47b93800(02515dca-bbf0-45e0-bd6e-e76f0c89bd0b) - lustre-MDT0000_UUID, 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b 00000020:00000080:1.0:1788572299.858143:0:160918:0:(genops.c:1385:class_connect()) connect: client 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b, cookie 0x8364c713e25bb2f6 00000100:00080000:1.0:1788572299.858156:0:160918:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:1.0:1788572299.858178:0:160918:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572299.858192:0:160918:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572299.858222:0:160918:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cea00 x1875444569487616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572299.858247:0:160918:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID 00800000:01000000:1.0:1788572299.858372:0:160918:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0000-mdc-ffff920a47b93800(02515dca-bbf0-45e0-bd6e-e76f0c89bd0b) successfully (5) 00800000:00000004:1.0:1788572299.858376:0:160918:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0000-mdc-ffff920a47b93800: age 8058, max_age 0 00000100:00080000:1.0:1788572299.858423:0:2398:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a528cc700 x1875444569487744/t0(0) o41->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00800000:01000000:1.0:1788572299.858457:0:160918:0:(lmv_obd.c:338:lmv_connect_mdc()) connect to lustre-MDT0001-mdc-ffff920a47b93800(02515dca-bbf0-45e0-bd6e-e76f0c89bd0b) - lustre-MDT0001_UUID, 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b 00000020:00000080:1.0:1788572299.858465:0:160918:0:(genops.c:1385:class_connect()) connect: client 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b, cookie 0x8364c713e25bb2fd 00000100:00080000:1.0:1788572299.858472:0:160918:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:1.0:1788572299.858478:0:160918:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572299.858484:0:160918:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572299.858495:0:160918:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cd500 x1875444569487872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572299.858504:0:160918:0:(pinger.c:373:ptlrpc_pinger_add_import()) adding pingable import 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID 00800000:01000000:1.0:1788572299.858558:0:160918:0:(lmv_obd.c:392:lmv_connect_mdc()) Connected to lustre-MDT0001-mdc-ffff920a47b93800(02515dca-bbf0-45e0-bd6e-e76f0c89bd0b) successfully (5) 00800000:00000004:1.0:1788572299.858561:0:160918:0:(obd_class.h:1056:obd_statfs_async()) lustre-MDT0001-mdc-ffff920a47b93800: age 8058, max_age 0 00000100:00080000:1.0:1788572299.858584:0:2397:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a528cce00 x1875444569488000/t0(0) o41->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000080:00000004:1.0:1788572299.858644:0:160918:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff920a47b93800: age 8058, max_age 9057 00800000:00000004:1.0:1788572299.858649:0:160918:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 8058, max_age 9057 00000100:00080000:1.0:1788572299.858666:0:160918:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a528cca80 x1875444569488128/t0(0) o41->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572299.858674:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5540 going to sleep for 0 seconds 00000100:00080000:3.0:1788572299.867377:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000000105e0b7): req@ffff920a528cc700 x1875444569487744/t0(0) o41->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572299.867403:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a34d5fe4): req@ffff920a528cca80 x1875444569488128/t0(0) o41->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572299.867418:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569487616:192.168.202.157@tcp:38:mount.lustre.0 00000100:00100000:3.0:1788572299.867446:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 0 gen 1 00000100:00080000:3.0:1788572299.867453:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff920a47b93800: connect to target with instance 5 00000100:00080000:3.0:1788572299.867460:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-MDT0001-mdc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xae7a7e7be344d3b8 00000100:00080000:3.0:1788572299.867468:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-MDT0001_UUID 00000100:00080000:3.0:1788572299.867470:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to FULL 00000100:00100000:1.0:1788572299.867762:0:160918:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5540 going to sleep for 11 seconds 00010000:00080000:1.0:1788572299.873386:0:2398:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff920a528cc700 x1875444569487744/t0(0) o41->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572315 ref 1 fl Rpc:RQU/200/0 rc 0/0 job:'mount.lustre.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572299.873398:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:0:1875444569487744:192.168.202.157@tcp:41:mount.lustre.0 00000100:00100000:1.0:1788572299.875286:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:0:1875444569488000:192.168.202.157@tcp:41:mount.lustre.0 00000100:00100000:3.0:1788572299.882242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569488384:192.168.202.157@tcp:8:mount.lustre.0 00000100:00100000:3.0:1788572299.882293:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46525180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569488512:192.168.202.157@tcp:8:mount.lustre.0 00010000:00010000:3.0:1788572301.726843:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572303.771769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572304.800276:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572305.378716:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569488896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572305.379220:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572305.379229:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569489024:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572305.382864:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569489152:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572305.382990:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:3.0:1788572305.382995:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572305.382999:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572305.383006:0:124398:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9058 00000100:00080000:3.0:1788572305.383013:0:124398:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572305.383031:0:124398:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572305.383042:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:3.0:1788572305.383059:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3c380 x1875444569489280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572305.399801:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569489408:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572305.402058:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569488896:192.168.202.157@tcp:400:kworker.0 00010000:00080000:3.0:1788572305.402986:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=61549) req@ffff920a51b3f100 x1875444569489024/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572321 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572305.403002:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569489024:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572305.454648:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569489152:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572305.454780:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569489408:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572305.820379:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572306.843895:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572306.951375:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572306.951514:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584333312 00000100:00100000:2.0:1788572306.951528:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584333312 00000100:00100000:2.0:1788572306.951635:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 0 00000100:00100000:2.0:1788572306.951648:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:194082:x1875444584333312:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572306.951674:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.157@tcp lock: ffff920a46ec6200/0x8364c713e25bb2e1 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bdc4d expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572306.951780:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572306.951789:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3e300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:194082:x1875444584333312:12345-192.168.202.157@tcp:104: Request processed in 143us (431us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572306.951796:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 0 00010000:00010000:1.0:1788572307.868234:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788572308.891873:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788572308.891887:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 335 10000000:01000000:1.0:1788572308.891896:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572308.891980:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572308.891994:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572308.892008:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572308.892047:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569489664:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572308.892911:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52bbea80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572308.918439:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569489664:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572308.918510:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572308.918528:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572308.918674:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569489792:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788572308.918822:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52bbea80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572308.928073:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569489792:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788572308.928154:0:160932:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1800 already not connected (conn 2, was 0): DISCONN 00010000:00080000:1.0:1788572308.928174:0:160932:0:(ldlm_lib.c:85:import_set_conn()) imp ffff920a51ac1800@lustre-OST0000-osc-ffff920a47b93800: found existing conn 192.168.202.157@tcp, moved to head 00000100:00080000:1.0:1788572308.928182:0:160932:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572308.928188:0:160932:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:1.0:1788572308.928200:0:160932:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572308.928215:0:160932:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572308.928260:0:160932:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cc700 x1875444569489920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788572308.928344:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572308.928358:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572308.928369:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572309.915830:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572310.492168:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:1.0:1788572310.492175:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572310.492179:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572310.492194:0:124398:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572310.492206:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572310.492225:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cd500 x1875444569490432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572310.492277:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569490048:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572310.492379:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490304:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572310.492417:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572310.492423:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490176:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572310.492446:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490560:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572310.493662:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569490432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572310.501257:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490560:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572310.501275:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569490048:192.168.202.157@tcp:400:kworker.0 00010000:00080000:1.0:1788572310.504034:0:2397:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a528cc700 x1875444569490176/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572326 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572310.504054:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490176:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572310.504068:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569490304:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572311.963836:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572314.015713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572315.619134:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 0 suppress 1 00000100:00080000:1.0:1788572315.619142:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572315.619147:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572315.619165:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572315.619179:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 134217728, lost: 0. 00000100:00100000:1.0:1788572315.619200:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cea00 x1875444569491072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572315.619275:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569490816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572315.619399:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569491200:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572315.619624:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569491072:192.168.202.157@tcp:8:kworker.0 00000100:00080000:2.0:1788572315.622940:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff920a47b93800: connect to target with instance 9 00000100:00080000:2.0:1788572315.622954:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572315.622970:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572315.622974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00020000:01000000:2.0:1788572315.622982:0:2396:0:(lov_obd.c:398:lov_set_osc_active()) Marking OSC lustre-OST0000_UUID active 00000080:00000004:2.0:1788572315.622987:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572315.622993:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00100000:2.0:1788572315.623004:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569491072:192.168.202.157@tcp:8:kworker.0 00010000:00080000:1.0:1788572315.623194:0:2398:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a528cc700 x1875444569490816/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572331 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572315.623209:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569490816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572315.623235:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569491200:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572316.060192:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572318.108157:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572318.314124:0:161129:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572318.314168:0:161129:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572318.314257:0:161129:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 161129 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572318.314271:0:161129:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161129 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572318.314282:0:161129:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161129 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572318.314521:0:161129:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job touch:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161129:1875444569491328:192.168.202.157@tcp:101:touch.0 00000100:00100000:3.0:1788572318.315115:0:161129:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eeab40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572318.324237:0:161129:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job touch:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161129:1875444569491328:192.168.202.157@tcp:101:touch.0 00010000:00010000:1.0:1788572318.324265:0:161129:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324277:0:161129:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324290:0:161129:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324307:0:161129:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324332:0:161129:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324340:0:161129:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324347:0:161129:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572318.324360:0:161129:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3f100 x1875444569491328/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'touch.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572318.324382:0:161129:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324395:0:161129:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572318.324401:0:161129:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572318.324429:0:161129:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:1.0:1788572318.324434:0:161129:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb320 00000080:00010000:1.0:1788572318.324438:0:161129:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572318.324442:0:161129:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb320 from it ffffaf7302fbbbf8 00010000:00010000:1.0:1788572318.324445:0:161129:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.324453:0:161129:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572318.324479:0:161129:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572318.324504:0:161129:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572318.324520:0:161129:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572318.324529:0:161129:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572318.324541:0:161129:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572318.324548:0:161129:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572318.324553:0:161129:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572318.324556:0:161129:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb320 00000080:00010000:1.0:1788572318.324560:0:161129:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb320 from it ffffaf7302fbbb38 00010000:00010000:1.0:1788572318.324564:0:161129:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572318.324572:0:161129:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000080:00200000:1.0:1788572318.324581:0:161129:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572318.324809:0:161129:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f74.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), file ffff920a455c9080, open_flags 8941, mode 81b6 opened 0 00000080:00200000:1.0:1788572318.324819:0:161129:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f74.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=open|creat 00000100:00100000:1.0:1788572318.324932:0:161129:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job touch:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161129:1875444569491456:192.168.202.157@tcp:700:touch.0 00000100:00100000:1.0:1788572318.325058:0:161129:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52bbea80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572318.332028:0:161129:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job touch:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161129:1875444569491456:192.168.202.157@tcp:700:touch.0 00000002:00010000:1.0:1788572318.332064:0:161129:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f74.replay-single,[0x200027501:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: open|creat flags 0104502 00010000:00010000:1.0:1788572318.332083:0:161129:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb320 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bde06 expref: -99 pid: 161129 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 1 00000100:00100000:3.0:1788572318.359919:0:161129:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job touch:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161129:1875444569491840:192.168.202.157@tcp:900:touch.0 00000100:00100000:3.0:1788572318.360007:0:161129:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572318.382505:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569492352:192.168.202.157@tcp:2:touch.0 00000100:00100000:1.0:1788572318.395072:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569492352:192.168.202.157@tcp:2:touch.0 00000080:00200000:2.0:1788572318.415158:0:161130:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572318.415203:0:161130:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572318.415271:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572318.415281:0:161130:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572318.415294:0:161130:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572318.415330:0:161130:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569492736:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572318.415428:0:161130:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46496780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572318.421208:0:161130:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569492736:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572318.421244:0:161130:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421256:0:161130:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421269:0:161130:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421280:0:161130:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421293:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421302:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421310:0:161130:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572318.421322:0:161130:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a52892a00 x1875444569492736/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572318.421336:0:161130:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421348:0:161130:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572318.421355:0:161130:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572318.421383:0:161130:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:1.0:1788572318.421390:0:161130:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb335 00000080:00010000:1.0:1788572318.421395:0:161130:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572318.421398:0:161130:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb335 from it ffffaf7302f9bb58 00010000:00010000:1.0:1788572318.421401:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421410:0:161130:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572318.421433:0:161130:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572318.421453:0:161130:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=f74.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:1.0:1788572318.421458:0:161130:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f74.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:1.0:1788572318.421480:0:161130:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f74.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572318.421523:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421529:0:161130:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.421539:0:161130:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161130 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572318.421559:0:161130:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569492864:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572318.421663:0:161130:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52bbea80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572318.430802:0:161130:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569492864:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572318.430832:0:161130:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430863:0:161130:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: --/PR res: [0x200027501:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430871:0:161130:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: --/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430882:0:161130:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430893:0:161130:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430906:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430915:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430922:0:161130:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572318.430933:0:161130:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a528cc000 x1875444569492864/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/640 e 0 to 0 dl 1788572334 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572318.430943:0:161130:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572318.430956:0:161130:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.430966:0:161130:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/1 (0 0) 00000080:00200000:1.0:1788572318.430973:0:161130:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572318.431050:0:161130:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x1:0x0](ffff920a50d45348): rc = 0 00000080:00200000:1.0:1788572318.431063:0:161130:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x1:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788572318.431067:0:161130:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x1:0x0](ffff920a50d45348) for lock 0x8364c713e25bb33c 00000080:00010000:1.0:1788572318.431078:0:161130:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x1:0x0](ffff920a50d45348) 00000080:00010000:1.0:1788572318.431082:0:161130:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb33c from it ffffaf7302f9bb20 00010000:00010000:1.0:1788572318.431085:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.431093:0:161130:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572318.431128:0:161130:0:(file.c:6296:ll_getattr_dentry()) START file f74.replay-single:[0x200027501:0x1:0x0](ffff920a50d45348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572318.431135:0:161130:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x1:0x0](ffff920a50d45348),name=f74.replay-single 00000002:00010000:1.0:1788572318.431156:0:161130:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572318.431165:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572318.431176:0:161130:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572318.431195:0:161130:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572318.431200:0:161130:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x1:0x0](ffff920a50d45348) 00000080:00010000:1.0:1788572318.431203:0:161130:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x1:0x0](ffff920a50d45348) for lock 0x8364c713e25bb33c 00000080:00010000:1.0:1788572318.431207:0:161130:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb33c from it ffffaf7302f9bcb8 00010000:00010000:1.0:1788572318.431210:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572318.431216:0:161130:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572318.431248:0:161130:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x1:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:1.0:1788572318.431257:0:161130:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200027501:0x1:0x0] 00000008:00010000:1.0:1788572318.431287:0:161130:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:1.0:1788572318.431291:0:161130:0:(osc_lock.c:1312:osc_lock_init()) ### lock ffff920a60b6bf30, osc lock ffff920a60bac390, flags 0x400001002 00010000:00010000:1.0:1788572318.431338:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->0] (req 0->0) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.431358:0:161130:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x400001000 ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 3/1,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572318.431405:0:161130:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1788572318.431416:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572318.431430:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569492992:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572318.440657:0:2397:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440685:0:2397:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 6/2,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440700:0:2397:0:(ldlm_resource.c:1750:ldlm_resource_add_lock()) ### About to add this lock ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440718:0:2397:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000008:00010000:1.0:1788572318.440735:0:2397:0:(osc_lock.c:187:osc_lock_lvb_update()) ### acquired size 0, setting rss=0; kms=0, end=18446744073709551615 ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 5/2,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00000020:00200000:1.0:1788572318.440760:0:2397:0:(cl_io.c:1849:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1788572318.440822:0:2397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 6/2,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440832:0:2397:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440841:0:2397:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 5/1,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572318.440851:0:2397:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 4/0,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572318.440873:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569492992:192.168.202.157@tcp:101:rm.0 00000080:00200000:3.0:1788572318.455485:0:161130:0:(file.c:1775:ll_merge_attr_nolock()) [0x200027501:0x1:0x0] updating i_size 0 i_blocks 0 00000020:00010000:3.0:1788572318.455498:0:161130:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff920a52505030 (ffff920a51965878/0) at cl_lock_release():245 00000020:00010000:3.0:1788572318.455503:0:161130:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a52505030 (ffff920a51965878/0) at cl_lock_cancel():142 00000020:00010000:3.0:1788572318.455508:0:161130:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff920a60b6bf30 (ffff920a519656c8/1) at cl_lock_cancel():142 00000020:00010000:3.0:1788572318.455515:0:161130:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a52505030 (ffff920a51965878/0) at cl_lock_fini():72 00000020:00010000:3.0:1788572318.455517:0:161130:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff920a60b6bf30 (ffff920a51965878/1) at cl_lock_fini():72 00000080:00200000:3.0:1788572318.455536:0:161130:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x1:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572318.455615:0:161130:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572318.455639:0:161130:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572318.455659:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572318.455673:0:161130:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572318.455681:0:161130:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572318.455689:0:161130:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572318.455692:0:161130:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb335 00000080:00010000:3.0:1788572318.455699:0:161130:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb335 from it ffffaf7302f9bc10 00010000:00010000:3.0:1788572318.455703:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572318.455713:0:161130:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572318.455732:0:161130:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572318.455754:0:161130:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f74.replay-single, flags=0 00000080:00200000:3.0:1788572318.455762:0:161130:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572318.455769:0:161130:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572318.455775:0:161130:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455783:0:161130:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572318.455789:0:161130:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572318.455793:0:161130:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572318.455795:0:161130:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb335 00000080:00010000:3.0:1788572318.455799:0:161130:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb335 from it ffffaf7302f9bd68 00010000:00010000:3.0:1788572318.455801:0:161130:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455808:0:161130:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:3.0:1788572318.455817:0:161130:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572318.455823:0:161130:0:(namei.c:2351:ll_unlink()) VFS Op:name=f74.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:3.0:1788572318.455851:0:161130:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572318.455876:0:161130:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572318.455886:0:161130:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x1:0x0]: losing layout lock 00010000:00010000:3.0:1788572318.455896:0:161130:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb32e lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x1:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bde37 expref: -99 pid: 161129 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572318.455903:0:161130:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x1:0x0]: losing layout lock 00010000:00010000:3.0:1788572318.455933:0:161130:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 1/0,0 mode: --/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455941:0:161130:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb32e lrc: 1/0,0 mode: --/CR res: [0x200027501:0x1:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bde37 expref: -99 pid: 161129 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455947:0:161130:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:3.0:1788572318.455952:0:161130:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:3.0:1788572318.455956:0:161130:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb33c lrc: 0/0,0 mode: --/PR res: [0x200027501:0x1:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bde61 expref: -99 pid: 161130 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455968:0:161130:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb32e lrc: 0/0,0 mode: --/CR res: [0x200027501:0x1:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bde37 expref: -99 pid: 161129 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572318.455983:0:161130:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817e00/0x8364c713e25bb335 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bde45 expref: -99 pid: 161130 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000020:00100000:3.0:1788572318.456001:0:161130:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572318.456015:0:161130:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ea00 x1875444569493248/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572318.456041:0:161130:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569493248:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572318.456135:0:161130:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572318.459250:0:161130:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161130:1875444569493248:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572318.459320:0:161130:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x1:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572318.459333:0:161130:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572318.459360:0:161130:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x1:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572318.459429:0:161130:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x1:0x0](ffff920a50d45348) 00000008:00010000:3.0:1788572318.459503:0:161130:0:(osc_object.c:202:osc_object_ast_clear()) ### update lock size 0 blocks 0 [cma]time: 1788572318 1788572318 1788572318 by oinfo size 0 blocks 0 [cma]time 1788572318 1788572318 1788572318 ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 1/0,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572320.164220:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572320.599502:0:161158:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161158:1875444569493504:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572320.599533:0:161158:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317241/353529 objects 1023639/1024000 00000080:00200080:3.0:1788572320.599677:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.599701:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54092c00 uarg=0000000000000000 00800000:00000004:3.0:1788572320.599710:0:161158:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 8058, max_age 9078 00000100:00100000:3.0:1788572320.599749:0:161158:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161158:1875444569493632:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572320.599809:0:161158:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eea3c0 going to sleep for 11 seconds 00800000:00000080:3.0:1788572320.608397:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608424:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608546:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608553:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608557:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608561:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608565:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608569:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608571:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608575:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608579:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608583:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608606:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608611:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608616:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608620:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608624:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608627:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608631:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608635:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00800000:00000080:3.0:1788572320.608638:0:161158:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00000080:00200080:3.0:1788572320.608642:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00020000:00000080:3.0:1788572320.608647:0:161158:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00020000:00000004:3.0:1788572320.608656:0:161158:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 8058, max_age 9078 00000100:00100000:3.0:1788572320.608775:0:161158:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161158:1875444569493760:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572320.608987:0:161158:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eeae40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572320.619460:0:161158:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161158:1875444569493760:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572320.619492:0:161158:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:3.0:1788572320.619564:0:161158:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc4d0d91e0 00020000:00000080:3.0:1788572320.619581:0:161158:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090000 uarg=0000000000000000 00020000:00000004:3.0:1788572320.619617:0:161158:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 8058, max_age 9078 00000100:00100000:3.0:1788572320.619651:0:161158:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161158:1875444569493888:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572320.619792:0:161158:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a46eeae40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572320.733375:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569494016:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572320.751340:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569494016:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572321.183600:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572322.203843:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572323.232077:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572324.251940:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572324.349891:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572324.349950:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584343936 00000100:00100000:2.0:1788572324.349964:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584343936 00000100:00100000:2.0:1788572324.350025:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 1 00000100:00100000:2.0:1788572324.350036:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a52893100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:41719:x1875444584343936:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572324.350059:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572324.350723:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572324.350739:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a52893100 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:LOV_OSC_UUID+4:41719:x1875444584343936:12345-192.168.202.157@tcp:104: Request processed in 701us (860us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572324.350748:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 1 00010000:00010000:1.0:1788572324.362664:0:160929:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572324.362685:0:160929:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 2/0,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x420000010000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572324.362705:0:160929:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79800 already unused, calling callback (ffffffffc11af300) 00010000:00010000:1.0:1788572324.362715:0:160929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 3/0,0 mode: PR/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x428400010000 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572324.362846:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 2/0,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572324.362854:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572324.362901:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 1/0,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572324.362911:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0000-osc-ffff920a47b93800 lock: 000000000893c4f5/0x8364c713e25bb343 lrc: 0/0,0 mode: --/PR res: [0x280000401:0x1a22:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x4c29400010020 nid: local remote: 0x265cfe11109bde68 expref: -99 pid: 161130 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572324.362941:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572324.362943:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572324.362945:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572324.362962:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572324.362987:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569494144:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572324.393155:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569494144:192.168.202.157@tcp:103:ldlm_bl.0 00010000:00010000:2.0:1788572325.278381:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572325.852815:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494400:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572325.852941:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572325.852977:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572325.852984:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569494272:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572325.853016:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494528:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572325.853037:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494784:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572325.858344:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494784:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572325.858368:0:2399:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a47b93800: committing for last_committed 38654705666 gen 1 00000100:00100000:3.0:1788572325.858400:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572325.864003:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569494272:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572325.864033:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494528:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572325.864044:0:2399:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383684 gen 1 00000100:00100000:3.0:1788572325.864086:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569494400:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572326.300084:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572327.324952:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572328.351482:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572328.740321:0:161747:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 21:38:47 (1788572327) 00000080:00200000:3.0:1788572328.797057:0:161750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572328.797093:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572328.797112:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572328.797128:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572328.797136:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572328.797144:0:161750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572328.797147:0:161750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb34a 00000080:00010000:3.0:1788572328.797154:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3b58 00010000:00010000:3.0:1788572328.797158:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572328.797166:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:3.0:1788572328.797185:0:161750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572328.797198:0:161750:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=13 00000080:00200000:3.0:1788572328.797203:0:161750:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572328.797215:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572328.797263:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.797271:0:161750:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.797280:0:161750:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572328.797308:0:161750:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161750:1875444569494912:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572328.797398:0:161750:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572328.811769:0:161750:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161750:1875444569494912:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572328.811796:0:161750:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811809:0:161750:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811817:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811823:0:161750:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811836:0:161750:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811842:0:161750:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec6400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572328.811846:0:161750:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811852:0:161750:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811859:0:161750:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811866:0:161750:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572328.811873:0:161750:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572328.811879:0:161750:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb351 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161750 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572328.811895:0:161750:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a51b3ea00 x1875444569494912/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572344 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572328.811910:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572328.811937:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572328.811949:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00010000:3.0:1788572328.811958:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3930 00010000:00010000:3.0:1788572328.811963:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:3.0:1788572328.811971:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572328.812041:0:161750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572328.812055:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572328.812062:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572328.812071:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:3.0:1788572328.812077:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572328.812082:0:161750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572328.812085:0:161750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb34a 00000080:00010000:3.0:1788572328.812090:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3be8 00010000:00010000:3.0:1788572328.812093:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:3.0:1788572328.812100:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572328.812108:0:161750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572328.812115:0:161750:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=16392 00000080:00200008:3.0:1788572328.812139:0:161750:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572328.812144:0:161750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572328.812151:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572328.812157:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572328.812165:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572328.812172:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572328.812176:0:161750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572328.812178:0:161750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb34a 00000080:00010000:3.0:1788572328.812181:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3cb8 00010000:00010000:3.0:1788572328.812184:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:3.0:1788572328.812192:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200008:3.0:1788572328.815189:0:161750:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572328.815200:0:161750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572328.815223:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572328.815240:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572328.815255:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572328.815263:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572328.815270:0:161750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572328.815274:0:161750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb34a 00000080:00010000:3.0:1788572328.815281:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3cb8 00010000:00010000:3.0:1788572328.815286:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572328.815294:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572328.815441:0:161750:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572328.815450:0:161750:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572328.815458:0:161750:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572328.820060:0:161750:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572328.820075:0:161750:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572328.820085:0:161750:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572328.820089:0:161750:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb34a 00000080:00010000:3.0:1788572328.820098:0:161750:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb34a from it ffffaf7300dd3bb0 00010000:00010000:3.0:1788572328.820102:0:161750:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572328.820112:0:161750:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572328.820126:0:161750:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572328.820137:0:161750:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00000080:00200080:3.0:1788572328.820173:0:161750:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c03066f0 arg=7ffc902a5d30 00000080:00200000:3.0:1788572328.820187:0:161750:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) name=d80a.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:3.0:1788572328.820222:0:161750:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572328.820281:0:161750:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572328.820288:0:161750:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572328.820291:0:161750:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb34a lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bde92 expref: -99 pid: 161158 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000020:00100000:3.0:1788572328.820316:0:161750:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572328.820329:0:161750:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ea00 x1875444569495168/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572328.820354:0:161750:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161750:1875444569495168:192.168.202.157@tcp:36:lfs.0 00000100:00100000:3.0:1788572328.820430:0:161750:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572328.838440:0:161750:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161750:1875444569495168:192.168.202.157@tcp:36:lfs.0 00000080:00200000:3.0:1788572328.838498:0:161750:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00200000:3.0:1788572328.838506:0:161750:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00200008:3.0:1788572328.838531:0:161750:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00010000:00010000:2.0:1788572329.372770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572329.812705:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495296:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572329.812730:0:161770:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812743:0:161770:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812756:0:161770:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812767:0:161770:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812779:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812787:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812795:0:161770:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572329.812806:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a46524700 x1875444569495296/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572345 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572329.812818:0:161770:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812831:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572329.812837:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572329.812863:0:161770:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572329.812868:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:3.0:1788572329.812873:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572329.812876:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf730165fb58 00010000:00010000:3.0:1788572329.812880:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812889:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572329.812913:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572329.812931:0:161770:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=29 00000080:00200000:3.0:1788572329.812935:0:161770:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:3.0:1788572329.812955:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788572329.812992:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.812999:0:161770:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.813008:0:161770:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572329.813028:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495424:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572329.813215:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af50c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572329.823308:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495424:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572329.823342:0:161770:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823360:0:161770:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823366:0:161770:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823375:0:161770:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823383:0:161770:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823392:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823397:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823403:0:161770:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572329.823416:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a51b3d880 x1875444569495424/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572345 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572329.823429:0:161770:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823440:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/2 (0 0) 00000080:00200000:3.0:1788572329.823446:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572329.823472:0:161770:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00200000:3.0:1788572329.823475:0:161770:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x2:0x0](ffff920a50d3cb08): rc = 0 00000080:00010000:3.0:1788572329.823478:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb35f 00000080:00010000:3.0:1788572329.823498:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00010000:3.0:1788572329.823503:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb35f from it ffffaf730165fa28 00010000:00010000:3.0:1788572329.823507:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823516:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572329.823552:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572329.823569:0:161770:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), flags=13 00000080:00200000:3.0:1788572329.823573:0:161770:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), intent=getattr 00000002:00010000:3.0:1788572329.823613:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572329.823651:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823659:0:161770:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572329.823668:0:161770:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572329.823690:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495552:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572329.823855:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af50c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572329.840358:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495552:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572329.840385:0:161770:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840400:0:161770:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840408:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840415:0:161770:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840424:0:161770:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840430:0:161770:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec7400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572329.840435:0:161770:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840442:0:161770:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840450:0:161770:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 5/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840457:0:161770:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572329.840464:0:161770:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840470:0:161770:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb366 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572329.840498:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a51b3d880 x1875444569495552/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572345 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572329.840519:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572329.840544:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/2 (0 0) 00000080:00200000:1.0:1788572329.840630:0:161770:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572329.840645:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572329.840653:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572329.840664:0:161770:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572329.840669:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572329.840674:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572329.840678:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:1.0:1788572329.840682:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf730165fbe8 00010000:00010000:1.0:1788572329.840686:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572329.840694:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572329.840707:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572329.840713:0:161770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=16408 00000080:00200000:1.0:1788572329.840719:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572329.840732:0:161770:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), flags=16392 00000080:00200000:1.0:1788572329.840736:0:161770:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), intent=getattr 00000002:00010000:1.0:1788572329.840748:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572329.840787:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840795:0:161770:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.840804:0:161770:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572329.840824:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495680:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572329.840904:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572329.852614:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495680:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572329.852639:0:161770:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852653:0:161770:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852659:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852666:0:161770:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852675:0:161770:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852681:0:161770:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572329.852687:0:161770:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852695:0:161770:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852703:0:161770:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 5/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852710:0:161770:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572329.852717:0:161770:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.852724:0:161770:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb36d lrc: 0/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572329.852744:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a528cc000 x1875444569495680/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572345 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572329.852759:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572329.852789:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/2 (0 0) 00000080:00200000:1.0:1788572329.852892:0:161770:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572329.852907:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572329.852917:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572329.852928:0:161770:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572329.852935:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572329.852940:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572329.852943:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:1.0:1788572329.852949:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf730165fb58 00010000:00010000:1.0:1788572329.852952:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572329.852965:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572329.852980:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572329.852986:0:161770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=13 00000080:00200008:1.0:1788572329.852994:0:161770:0:(file.c:6296:ll_getattr_dentry()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572329.852998:0:161770:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08),name=d80a.replay-single 00000002:00010000:1.0:1788572329.853010:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x2:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572329.853014:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/2 (0 0) 00010000:00010000:1.0:1788572329.853052:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.853058:0:161770:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.853067:0:161770:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572329.853088:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495808:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572329.853163:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572329.862382:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569495808:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572329.862411:0:161770:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862428:0:161770:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862441:0:161770:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862453:0:161770:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862465:0:161770:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862478:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862486:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862506:0:161770:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572329.862518:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a528cc000 x1875444569495808/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572345 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572329.862532:0:161770:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862542:0:161770:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/2 (0 0) 00000080:00200000:1.0:1788572329.862549:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572329.862577:0:161770:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00010000:1.0:1788572329.862583:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb374 00000080:00010000:1.0:1788572329.862624:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00010000:1.0:1788572329.862627:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb374 from it ffffaf730165fcb8 00010000:00010000:1.0:1788572329.862631:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.862642:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572329.863159:0:161770:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572329.863166:0:161770:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572329.863184:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572329.863195:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572329.863207:0:161770:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572329.863215:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572329.863219:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572329.863222:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:1.0:1788572329.863227:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf730165fcb8 00010000:00010000:1.0:1788572329.863230:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572329.863238:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572329.863341:0:161770:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572329.863349:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572329.863357:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572329.863365:0:161770:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572329.863372:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572329.863375:0:161770:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572329.863378:0:161770:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:1.0:1788572329.863381:0:161770:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf730165fbf8 00010000:00010000:1.0:1788572329.863385:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572329.863392:0:161770:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:1.0:1788572329.863401:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572329.863407:0:161770:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=267 00000080:00200000:1.0:1788572329.863412:0:161770:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572329.863418:0:161770:0:(file.c:1090:ll_file_open()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), flags 300000 00000002:00010000:1.0:1788572329.863429:0:161770:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x2:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788572329.863468:0:161770:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.863476:0:161770:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572329.863486:0:161770:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572329.863508:0:161770:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc000 x1875444569496064/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572329.863518:0:161770:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572329.863538:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496064:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572329.863704:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572329.870419:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496064:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572329.870442:0:161770:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870456:0:161770:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870463:0:161770:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870470:0:161770:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870479:0:161770:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870485:0:161770:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572329.870502:0:161770:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870513:0:161770:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870520:0:161770:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870526:0:161770:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572329.870533:0:161770:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572329.870540:0:161770:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb37b lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 161770 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572329.870556:0:161770:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572345 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572329.870571:0:161770:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572345 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572329.870617:0:161770:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572329.870648:0:161770:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000002:00100000:1.0:1788572329.870666:0:161770:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572345 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572329.870746:0:161770:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) cmd=c03066f0 arg=7ffc46d11ab0 00000080:00200000:1.0:1788572329.870761:0:161770:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00000100:00100000:1.0:1788572329.870815:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496192:192.168.202.157@tcp:700:lfs.0 00000100:00100000:1.0:1788572329.870888:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572329.874334:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496192:192.168.202.157@tcp:700:lfs.0 00000020:00100000:1.0:1788572329.874404:0:161770:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572329.874413:0:161770:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc700 x1875444569496448/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 560/520 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572329.874436:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496448:192.168.202.157@tcp:36:lfs.0 00000100:00100000:1.0:1788572329.874511:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572329.882894:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572329.887763:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584346752 00000100:00100000:3.0:1788572329.887780:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584346752 00000100:00100000:3.0:1788572329.888012:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 2 00000100:00100000:3.0:1788572329.888022:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a52892d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:189111:x1875444584346752:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572329.888041:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb374 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bdf25 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572329.888133:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572329.888140:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a52892d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:189111:x1875444584346752:12345-192.168.202.157@tcp:104: Request processed in 120us (5256us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572329.888147:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 2 00010000:00010000:2.0:1788572330.397335:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572330.721395:0:161779:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572330.721410:0:161779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572330.721494:0:161779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572330.721543:0:161779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572330.721559:0:161779:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00000080:00200000:2.0:1788572330.721568:0:161779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572330.721575:0:161779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572330.721579:0:161779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:2.0:1788572330.722333:0:161779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301cffcb8 00010000:00010000:2.0:1788572330.722341:0:161779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 1 00010000:00010000:2.0:1788572330.722352:0:161779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200008:2.0:1788572330.724750:0:161779:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572330.724762:0:161779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572330.724785:0:161779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572330.724805:0:161779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572330.724819:0:161779:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:2.0:1788572330.724827:0:161779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572330.724835:0:161779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572330.724838:0:161779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:2.0:1788572330.724845:0:161779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301cffcb8 00010000:00010000:2.0:1788572330.724849:0:161779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:2.0:1788572330.724857:0:161779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572330.724992:0:161779:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572330.725001:0:161779:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572330.725008:0:161779:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572330.725016:0:161779:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:2.0:1788572330.725022:0:161779:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572330.725026:0:161779:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572330.725028:0:161779:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:2.0:1788572330.725031:0:161779:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301cffbb0 00010000:00010000:2.0:1788572330.725034:0:161779:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:2.0:1788572330.725040:0:161779:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:2.0:1788572330.725049:0:161779:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572330.725057:0:161779:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:2.0:1788572330.727314:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:2.0:1788572330.727341:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46ef6800 uarg=0000000000000000 00800000:00000004:2.0:1788572330.727349:0:161779:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9079, max_age 9088 00000100:00100000:2.0:1788572330.727392:0:161779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496704:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572330.734151:0:161779:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a5240d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572330.751622:0:161779:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383686 gen 1 00000100:00100000:1.0:1788572330.751654:0:161779:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572345 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572330.751669:0:161779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496704:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572330.751687:0:161779:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317240/353529 objects 1023637/1024000 00000080:00200080:1.0:1788572330.751771:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.751790:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00800000:00000004:1.0:1788572330.751798:0:161779:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9079, max_age 9088 00000100:00100000:1.0:1788572330.751834:0:161779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496832:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572330.751910:0:161779:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481730c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572330.767178:0:161779:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 21474837476 gen 1 00000100:00100000:1.0:1788572330.767270:0:161779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496832:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572330.767295:0:161779:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00200080:1.0:1788572330.767372:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767390:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767399:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767403:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767408:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767413:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767417:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767421:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767424:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767502:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767510:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767526:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767532:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767536:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767540:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767543:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767547:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767551:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767555:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767558:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767562:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767565:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767569:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767572:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767576:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.767579:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.767583:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768782:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768799:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768803:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768809:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768813:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768817:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768821:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768825:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768829:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768832:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768836:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768840:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768843:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768847:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768850:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768854:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768856:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768861:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768864:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768867:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768871:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768875:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768878:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768882:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768885:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768890:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768893:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768898:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768901:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768905:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768908:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768912:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768915:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768918:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768921:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768938:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768942:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768946:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768949:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768953:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768957:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768961:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768964:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768968:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768972:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768975:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768979:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768982:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768985:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768989:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.768992:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.768996:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769000:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769004:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769008:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769011:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769014:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769019:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769022:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769026:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769029:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769033:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769036:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769039:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769042:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769046:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769049:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769053:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769056:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769060:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769063:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769067:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769070:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769074:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769077:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769081:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769085:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769088:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769092:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769095:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769098:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769102:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769105:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769109:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769112:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769115:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769119:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769123:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769126:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769130:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769134:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769137:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769141:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769144:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769147:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769150:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769153:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769157:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00800000:00000080:1.0:1788572330.769160:0:161779:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.769164:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.769170:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00020000:00000004:1.0:1788572330.769177:0:161779:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9079, max_age 9088 00000100:00100000:1.0:1788572330.769221:0:161779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496960:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572330.769313:0:161779:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481730c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572330.797434:0:161779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569496960:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572330.797464:0:161779:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261712/262144 00000080:00200080:1.0:1788572330.797544:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.797561:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00020000:00000004:1.0:1788572330.797569:0:161779:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9079, max_age 9088 00000100:00100000:1.0:1788572330.797644:0:161779:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569497088:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572330.797814:0:161779:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481730c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572330.817970:0:161779:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161779:1875444569497088:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572330.818001:0:161779:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:1.0:1788572330.818076:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818095:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818102:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818107:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818111:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818114:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818118:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818121:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818124:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818127:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818131:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818134:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818137:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818140:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818144:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818147:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818150:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818153:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818157:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818160:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818163:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818166:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818170:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818174:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818177:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818181:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818195:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818199:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818203:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818207:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818210:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818214:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818218:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818221:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818225:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818228:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818232:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818236:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818239:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818244:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818248:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818251:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818255:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818258:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818262:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818265:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818269:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818272:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818276:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818279:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818283:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818286:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818290:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818293:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818297:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818300:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818303:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818306:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818310:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818313:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818317:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818320:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818324:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818328:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818331:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818335:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818338:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818341:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818345:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818348:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818352:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818354:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818358:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818361:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818365:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818369:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818373:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818377:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818381:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818384:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818388:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818391:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818395:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818398:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818402:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818405:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818409:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818412:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818416:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818420:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818424:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818440:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818445:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818449:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818453:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818456:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818461:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818464:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818468:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818471:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818475:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818479:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818483:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818487:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818490:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818494:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818498:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818501:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818505:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818509:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818513:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818527:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818532:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818536:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818541:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818545:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818549:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818553:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818557:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818561:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818565:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818569:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818573:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818577:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200080:1.0:1788572330.818582:0:161779:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffc674fad50 00020000:00000080:1.0:1788572330.818635:0:161779:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900400 uarg=0000000000000000 00000080:00200008:1.0:1788572330.818652:0:161779:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:3.0:1788572330.905199:0:161786:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.905229:0:161786:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.905249:0:161786:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572330.905265:0:161786:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572330.905274:0:161786:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.905282:0:161786:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.905285:0:161786:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:3.0:1788572330.905293:0:161786:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301a1fbd0 00010000:00010000:3.0:1788572330.905298:0:161786:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572330.905307:0:161786:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572330.905325:0:161786:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572330.905340:0:161786:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=1536 00000080:00200000:3.0:1788572330.905345:0:161786:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.905351:0:161786:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.905358:0:161786:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572330.905367:0:161786:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572330.905373:0:161786:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.905377:0:161786:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.905379:0:161786:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:3.0:1788572330.905382:0:161786:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301a1fcd0 00010000:00010000:3.0:1788572330.905385:0:161786:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572330.905392:0:161786:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572330.905400:0:161786:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572330.905406:0:161786:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.905411:0:161786:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.905416:0:161786:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572330.905424:0:161786:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572330.905431:0:161786:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.905434:0:161786:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.905437:0:161786:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb358 00000080:00010000:3.0:1788572330.905440:0:161786:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb358 from it ffffaf7301a1fdb0 00010000:00010000:3.0:1788572330.905442:0:161786:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572330.905448:0:161786:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572330.905455:0:161786:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572330.905460:0:161786:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=33188, excl=1 00000080:00200000:3.0:1788572330.905463:0:161786:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) mode 100644 dev 0 00010000:00010000:3.0:1788572330.905487:0:161786:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572330.905545:0:161786:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572330.905553:0:161786:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572330.905555:0:161786:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb358 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bded1 expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:3.0:1788572330.905581:0:161786:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572330.905848:0:161786:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3fb80 x1875444569497344/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572330.905873:0:161786:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161786:1875444569497344:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572330.906019:0:161786:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572330.912104:0:161786:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161786:1875444569497344:192.168.202.157@tcp:36:mcreate.0 00000080:00200000:3.0:1788572330.912150:0:161786:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x3:0x0](ffff920a50d3e3c8) 00000080:00200000:3.0:1788572330.912188:0:161786:0:(namei.c:2070:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, unhashed 1 00000080:00200000:3.0:1788572330.932316:0:161789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.932351:0:161789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.932421:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.932433:0:161789:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.932445:0:161789:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572330.932473:0:161789:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497472:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572330.935666:0:161789:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572330.947558:0:161789:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497472:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572330.947627:0:161789:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947640:0:161789:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947652:0:161789:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947664:0:161789:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947677:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947689:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947697:0:161789:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572330.947707:0:161789:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3df80 x1875444569497472/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572346 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572330.947721:0:161789:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947734:0:161789:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572330.947740:0:161789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572330.947768:0:161789:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572330.947775:0:161789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb382 00000080:00010000:3.0:1788572330.947780:0:161789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.947783:0:161789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb382 from it ffffaf7301c1fb58 00010000:00010000:3.0:1788572330.947787:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947796:0:161789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572330.947822:0:161789:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572330.947843:0:161789:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:3.0:1788572330.947847:0:161789:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572330.947865:0:161789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572330.947898:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947902:0:161789:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.947909:0:161789:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 161789 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572330.947922:0:161789:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497600:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572330.947972:0:161789:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572330.951023:0:161789:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497600:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572330.951042:0:161789:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951057:0:161789:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951063:0:161789:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951070:0:161789:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951076:0:161789:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951085:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951090:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951095:0:161789:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572330.951102:0:161789:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3df80 x1875444569497600/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572346 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572330.951111:0:161789:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951117:0:161789:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/3 (0 0) 00000080:00200000:3.0:1788572330.951122:0:161789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572330.951138:0:161789:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x3:0x0], updating i_size 0 00000080:00200000:3.0:1788572330.951140:0:161789:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x3:0x0](ffff920a50d3e3c8): rc = 0 00000080:00200000:3.0:1788572330.951153:0:161789:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x3:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572330.951157:0:161789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x3:0x0](ffff920a50d3e3c8) for lock 0x8364c713e25bb389 00000080:00010000:3.0:1788572330.951164:0:161789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x3:0x0](ffff920a50d3e3c8) 00000080:00010000:3.0:1788572330.951166:0:161789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb389 from it ffffaf7301c1fb20 00010000:00010000:3.0:1788572330.951168:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572330.951174:0:161789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572330.951208:0:161789:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x200027501:0x3:0x0](ffff920a50d3e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572330.951215:0:161789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x3:0x0](ffff920a50d3e3c8),name=fsa-oleg257-client.virtnet 00000002:00010000:3.0:1788572330.951235:0:161789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x3:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.951244:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572330.951256:0:161789:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572330.951260:0:161789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.951264:0:161789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x3:0x0](ffff920a50d3e3c8) 00000080:00010000:3.0:1788572330.951266:0:161789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x3:0x0](ffff920a50d3e3c8) for lock 0x8364c713e25bb389 00000080:00010000:3.0:1788572330.951269:0:161789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb389 from it ffffaf7301c1fcb8 00010000:00010000:3.0:1788572330.951271:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572330.951276:0:161789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572330.951333:0:161789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.951341:0:161789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.951347:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572330.951354:0:161789:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572330.951360:0:161789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.951364:0:161789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.951367:0:161789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb382 00000080:00010000:3.0:1788572330.951371:0:161789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb382 from it ffffaf7301c1fc10 00010000:00010000:3.0:1788572330.951374:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572330.951381:0:161789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572330.951387:0:161789:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572330.951394:0:161789:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:3.0:1788572330.951397:0:161789:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572330.951401:0:161789:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572330.951405:0:161789:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572330.951409:0:161789:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572330.951413:0:161789:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572330.951415:0:161789:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572330.951416:0:161789:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb382 00000080:00010000:3.0:1788572330.951418:0:161789:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb382 from it ffffaf7301c1fd68 00010000:00010000:3.0:1788572330.951420:0:161789:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572330.951424:0:161789:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572330.951432:0:161789:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572330.951437:0:161789:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:3.0:1788572330.951458:0:161789:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572330.951478:0:161789:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572330.951488:0:161789:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x3:0x0]: losing layout lock 00010000:00010000:3.0:1788572330.951517:0:161789:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572330.951534:0:161789:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572330.951538:0:161789:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572330.951540:0:161789:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7400/0x8364c713e25bb389 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bdf87 expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572330.951551:0:161789:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb382 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bdf6b expref: -99 pid: 161789 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572330.951563:0:161789:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572330.951569:0:161789:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3df80 x1875444569497856/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572330.951739:0:161789:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497856:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572330.951838:0:161789:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45af5cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572330.960305:0:161789:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161789:1875444569497856:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572330.960380:0:161789:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572330.960391:0:161789:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572330.960398:0:161789:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x3:0x0](ffff920a50d3e3c8) 00000100:00100000:1.0:1788572330.979316:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569497984:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572332.446295:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572334.495584:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572336.092049:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569498112:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.092151:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498368:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.092180:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498624:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.092210:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572336.092228:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498240:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.092254:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498496:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.106356:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498368:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.106379:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498624:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.106412:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498240:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.106423:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569498496:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572336.106432:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569498112:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572336.539958:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572337.911856:0:161861:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:2.0:1788572338.589076:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572340.640257:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572341.212204:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 24 seconds 00000100:00100000:2.0:1788572341.212302:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569498880:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572341.212302:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569498752:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572341.212387:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499008:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572341.216709:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569499136:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572341.223904:0:2400:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff920a47b93800 of lustre-MDT0000_UUID@192.168.202.157@tcp abruptly disconnected: reconnecting 00000100:00080000:2.0:1788572341.223913:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1788572341.223918:0:2400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff920a47b93800: Connection to lustre-MDT0000 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:3.0:1788572341.224627:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499264:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572341.227272:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499008:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572341.233163:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499264:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572341.256221:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572341.256239:0:2400:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9058 00000100:00080000:2.0:1788572341.256256:0:2400:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572341.256269:0:2400:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572341.256303:0:2400:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569499392/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572341.256336:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569498880:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572341.256383:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569499136:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572341.256649:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569499392:192.168.202.157@tcp:38:ptlrpcd_01_01.0 00000100:00080000:2.0:1788572341.260964:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572341.260979:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572341.260984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569499392:192.168.202.157@tcp:38:ptlrpcd_01_01.0 00010000:00010000:3.0:1788572342.684478:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572344.731710:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572346.332076:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572346.332085:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572346.332090:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572346.332101:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9099 00000100:00080000:2.0:1788572346.332113:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788572346.332121:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572346.332134:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569499520:192.168.202.157@tcp:400:kworker.0 00000008:00100000:2.0:1788572346.332147:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572346.332174:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569499648/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572346.332292:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572346.332306:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569499904:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572346.332398:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569500032:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572346.339389:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569499776:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572346.354035:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569499904:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572346.354084:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569500032:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572346.779825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572348.830388:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572350.877671:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572351.457151:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572351.457160:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572351.457165:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572351.457177:0:124398:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9104 00000100:00080000:1.0:1788572351.457191:0:124398:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572351.457200:0:124398:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572351.457216:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572351.457238:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528ce300 x1875444569500288/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572351.457285:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572351.457288:0:124398:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff920a47b93800: disconnect after 21s idle 00000100:00080000:1.0:1788572351.457297:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572351.457300:0:124398:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff920a47b93800: disconnect after 21s idle 00000100:00100000:1.0:1788572351.457346:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569500416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572351.457469:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572351.457476:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569500160:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572351.458202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500288:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572351.458264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500544:192.168.202.157@tcp:9:kworker.0 00000100:00100000:2.0:1788572351.458294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500672:192.168.202.157@tcp:9:kworker.0 00000100:00080000:2.0:1788572351.462208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572351.462228:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572351.462235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500288:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572351.462264:0:2396:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff920a528cd180 x1875444569500544/t0(0) o9->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572361 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572351.462275:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572351.462279:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500544:192.168.202.157@tcp:9:kworker.0 00000100:00080000:2.0:1788572351.462293:0:2396:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff920a528cfb80 x1875444569500672/t0(0) o9->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572361 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572351.462300:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:2.0:1788572351.462303:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500672:192.168.202.157@tcp:9:kworker.0 00000100:00100000:1.0:1788572351.462415:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569500416:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572352.928940:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572354.972295:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572356.577410:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572356.577417:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572356.577421:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572356.577432:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9109 00000100:00080000:1.0:1788572356.577441:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:1.0:1788572356.577448:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572356.577458:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572356.577475:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528ce680 x1875444569500928/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572356.577525:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569501056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572356.577630:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572356.577637:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569500800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572356.577687:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500928:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572356.598611:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572356.598627:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572356.598632:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569500928:192.168.202.157@tcp:38:kworker.0 00010000:00010000:2.0:1788572357.019837:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788572357.596318:0:2399:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572341/real 1788572341] req@ffff920a52890380 x1875444569498752/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572357 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572357.596353:0:2399:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52890380 x1875444569498752/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572357 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572357.596363:0:2399:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from FULL to DISCONN 00000100:02020000:3.0:1788572357.596366:0:2399:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.157@tcp: Connection to MGS (at 192.168.202.157@tcp) was lost; in progress operations using this service will fail 00000100:00080000:3.0:1788572357.596370:0:2399:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.157@tcp for MGC192.168.202.157@tcp not replayable, auto-deactivating 00000100:00080000:3.0:1788572357.596372:0:2399:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:3.0:1788572357.596388:0:2399:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a52890380 x1875444569498752/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572357 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572357.596397:0:2399:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a52892300 x1875444569499520/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572362 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572357.596420:0:2399:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a528cd880 x1875444569500160/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572367 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572357.596972:0:2399:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a528cf800 x1875444569500800/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572372 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:3.0:1788572357.596984:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:3.0:1788572357.596987:0:2399:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:3.0:1788572357.596989:0:2399:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:3.0:1788572357.596991:0:2399:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:3.0:1788572357.597018:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569498752:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572357.597235:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569499520:192.168.202.157@tcp:400:kworker.0 00000100:00080000:1.0:1788572357.597353:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572357.597358:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572357.597361:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572357.597367:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9058 00000100:00080000:1.0:1788572357.597391:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572357.597495:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cdf80 x1875444569501184/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572357.597523:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572357.597526:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572357.597528:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572357.597532:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9115 00000100:00080000:1.0:1788572357.597537:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572357.597542:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572357.597552:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572357.597565:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528ced80 x1875444569501312/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572357.598090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569501184:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572357.598122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501312:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572357.604668:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572357.604684:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572357.604689:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501312:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572359.070761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572361.116648:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572362.717177:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572362.717188:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572362.717191:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572362.717195:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572362.717200:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572362.717212:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9116 00000100:00080000:1.0:1788572362.717222:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572362.717231:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572362.717244:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572362.717279:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cf100 x1875444569501568/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572362.717373:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569501696:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572362.717404:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501568:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572362.726893:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572362.726908:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572362.726914:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501568:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572362.730580:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569501696:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572363.163804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572365.211860:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572365.276115:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 25 seconds 00010000:00010000:1.0:1788572367.262132:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572367.836702:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572357/real 1788572357] req@ffff920a528cdf80 x1875444569501184/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572367 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572367.836736:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a528cdf80 x1875444569501184/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572367 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572367.836757:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572367.836783:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572367.836787:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569501184:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572367.837147:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572367.837153:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572367.837156:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572367.837162:0:124398:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9116 00000100:00080000:1.0:1788572367.837172:0:124398:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 5s 00000100:00080000:1.0:1788572367.837307:0:124398:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572367.837341:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cdc00 x1875444569501824/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572367.837366:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572367.837370:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572367.837372:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572367.837376:0:124398:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9121 00000100:00080000:1.0:1788572367.837380:0:124398:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572367.837384:0:124398:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572367.837394:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572367.837405:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528ced80 x1875444569501952/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572367.837454:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572367.837477:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569502080:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572367.842472:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569501824:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572367.842574:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501952:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572367.849383:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.157@tcp: connect to target with instance 0 10000000:01000000:3.0:1788572367.849400:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:3.0:1788572367.849409:0:2396:0:(import.c:1328:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.157@tcp) after server handle changed from 0x265cfe11109bdc38 to 0x265cfe11109bdfc6 00000100:00080000:3.0:1788572367.849414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:3.0:1788572367.849420:0:2396:0:(import.c:1718:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.157@tcp; invalidating 00000100:00100000:2.0:1788572367.851327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569501824:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572367.851365:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572367.851370:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572367.851375:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569501952:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572367.851481:0:162084:0:(import.c:1642:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.157@tcp to MGS@192.168.202.157@tcp 00000100:00100000:3.0:1788572367.851486:0:162084:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:3.0:1788572367.851489:0:162084:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:3.0:1788572367.851507:0:162084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572367.851519:0:162084:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572367.851527:0:162084:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572367.851552:0:162084:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572367.851563:0:162084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a46ec7000/0x8364c713e25bb2a9 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdc3f expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572367.851579:0:162084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572367.851633:0:162084:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572367.851639:0:162084:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572367.851643:0:162084:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572367.851656:0:162084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a46ec7a00/0x8364c713e25bb2b0 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdc46 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572367.851673:0:162084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572367.851681:0:162084:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572367.851687:0:162084:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:3.0:1788572367.851690:0:162084:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572367.851694:0:162084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb319 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdd11 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572367.851708:0:162084:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:3.0:1788572367.851720:0:162084:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:3.0:1788572367.851726:0:162084:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:3.0:1788572367.851729:0:162084:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572367.851732:0:162084:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a46ec7600/0x8364c713e25bb2e8 lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdc54 expref: -99 pid: 160918 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:3.0:1788572367.851746:0:162084:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:3.0:1788572367.851751:0:162084:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from RECOVER to FULL 10000000:01000000:3.0:1788572367.851754:0:162084:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:3.0:1788572367.851757:0:162084:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:3.0:1788572367.851761:0:162084:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) MGC192.168.202.157@tcp: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00100000:1.0:1788572367.855249:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569502080:192.168.202.157@tcp:400:kworker.0 10000000:01000000:1.0:1788572369.051893:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788572369.051903:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 336 10000000:01000000:1.0:1788572369.051911:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572369.051985:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.051998:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.052010:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572369.052047:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502208:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.052166:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481733c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572369.058030:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502208:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572369.058320:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.058336:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572369.058519:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502336:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.058683:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481733c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572369.069291:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502336:192.168.202.157@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788572369.069419:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:1.0:1788572369.069435:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.069450:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:1.0:1788572369.069462:0:160932:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:1.0:1788572369.083883:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788572369.083896:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff920a47b93800 from 52 10000000:01000000:1.0:1788572369.083902:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572369.083970:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.083982:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.083992:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572369.084022:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502464:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.084357:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481733c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572369.098500:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502464:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572369.098532:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfd4 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.098545:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bdfd4 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572369.098681:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502592:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.098769:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481733c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572369.129850:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502592:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.130018:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502720:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572369.130120:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a481733c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572369.136014:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569502720:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000040:00080000:2.0:1788572369.137354:0:162085:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000040:00080000:2.0:1788572369.159874:0:162086:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000100:00100000:2.0:1788572369.190915:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525180 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569503360:192.168.202.157@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788572369.191281:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572369.191292:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a51a71800/0x8364c713e25bb3a5 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfe2 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572369.191304:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a51a71800/0x8364c713e25bb3a5 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bdfe2 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572369.307902:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572370.479227:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572370.479725:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584364544 00000100:00100000:3.0:1788572370.479736:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584364544 00000100:00100000:3.0:1788572370.479802:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 3 00000100:00100000:3.0:1788572370.479814:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a52892300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:196584:x1875444584364544:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572370.479833:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572370.479922:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572370.479930:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a52892300 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:196584:x1875444584364544:12345-192.168.202.157@tcp:104: Request processed in 119us (714us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572370.479938:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 3 00010000:00010000:1.0:1788572370.480091:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572370.480100:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572370.480108:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79200 already unused, calling callback (ffffffffc1336c20) 10000000:00010000:1.0:1788572370.480113:0:160930:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572370.480122:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572370.480130:0:160930:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572370.480136:0:160930:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572370.480179:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572370.480186:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572370.480206:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572370.480214:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79200/0x8364c713e25bb390 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bdfcd expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572370.480235:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572370.480238:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572370.480240:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572370.480255:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569503488:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572370.496920:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569503488:192.168.202.157@tcp:103:ldlm_bl.0 10000000:01000000:2.0:1788572372.332784:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788572372.332793:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 336 10000000:01000000:2.0:1788572372.332798:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572372.332853:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572372.332870:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572372.332883:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572372.332915:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569503616:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572372.333019:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572372.372727:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569503872:192.168.202.157@tcp:38:ll_cfg_requeue.0 00010000:00010000:1.0:1788572372.379840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:2.0:1788572372.394175:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a46524000 x1875444569503872/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572382 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572372.394201:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff920a47b93800: connect to target with instance 81 00000100:00080000:2.0:1788572372.394233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:2.0:1788572372.394409:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572372.394413:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383686 gen 1 00000100:00100000:2.0:1788572372.394420:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3fb80 x1875444569497344/t343597383688(343597383688) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572346 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572372.394429:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 343597383686 last 0 00000100:00080000:2.0:1788572372.394437:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572345 ref 1 fl New:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:2.0:1788572372.394452:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569503872:192.168.202.157@tcp:38:ll_cfg_requeue.0 00000100:00100000:2.0:1788572372.394480:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569496064:192.168.202.157@tcp:101:lfs.0 00000100:00080000:2.0:1788572372.415141:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572388 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00080000:2.0:1788572372.415167:0:2396:0:(mdc_request.c:696:mdc_replay_open()) updating handle from 0x265cfe11109bdf33 to 0x265cfe11109be1d3 00000100:00020000:2.0:1788572372.415172:0:2396:0:(client.c:3449:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572388 ref 2 fl Interpret:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00080000:2.0:1788572372.454845:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:2.0:1788572372.454855:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383686 gen 1 00000100:00100000:2.0:1788572372.454873:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3fb80 x1875444569497344/t343597383688(343597383688) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572346 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572372.454884:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 343597383686 last 343597383686 00000100:00080000:2.0:1788572372.454893:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3fb80 x1875444569497344/t343597383688(343597383688) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572346 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572372.454908:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569496064:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572372.454937:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569497344:192.168.202.157@tcp:36:mcreate.0 00000100:00080000:2.0:1788572372.956007:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572372.956015:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788572372.956106:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569504000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572372.956224:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572372.956232:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569504128:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572372.960803:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569504000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572372.965541:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569504128:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572374.432517:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572376.058475:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a51b3fb80 x1875444569497344/t343597383688(343597383688) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572388 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572376.058492:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572376.058495:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383686 gen 1 00000100:00100000:3.0:1788572376.058500:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3fb80 x1875444569497344/t343597383688(343597383688) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572388 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572376.058509:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 343597383686 last 343597383688 00000100:00080000:3.0:1788572376.058516:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3df80 x1875444569497856/t343597383689(343597383689) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572346 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572376.058529:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569497344:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572376.058547:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569497856:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572376.076951:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383688 gen 1 00000100:00100000:3.0:1788572376.076981:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3df80 x1875444569497856/t343597383689(343597383689) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572392 ref 2 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572376.076994:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a51b3df80 x1875444569497856/t343597383689(343597383689) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572392 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572376.077001:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572376.077004:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 343597383688 gen 1 00000100:00100000:3.0:1788572376.077008:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3df80 x1875444569497856/t343597383689(343597383689) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572392 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572376.077014:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 343597383688 last 343597383689 00000100:00080000:3.0:1788572376.077018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:2.0:1788572376.093418:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569497856:192.168.202.157@tcp:36:rm.0 00010000:00080000:3.0:1788572376.093528:0:162133:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff920a47b93800 to lustre-MDT0000_UUID@192.168.202.157@tcp 00010000:00010000:3.0:1788572376.093533:0:162133:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff920a47b93800 (1) 00010000:00010000:3.0:1788572376.093578:0:162133:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb35f lrc: 0/0,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x265cfe11109bdeed expref: -99 pid: 161770 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572376.093629:0:162133:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff920a47b93800 00000100:00080000:3.0:1788572376.094939:0:162133:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:1.0:1788572376.170768:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569496448:192.168.202.157@tcp:36:lfs.0 00000080:00200000:1.0:1788572376.170832:0:161770:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00200000:1.0:1788572376.170840:0:161770:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000080:00200008:1.0:1788572376.170879:0:161770:0:(file.c:479:ll_file_release()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), flags 300000 00000002:00100000:1.0:1788572376.170929:0:161770:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572388 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572376.170941:0:161770:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572376.170947:0:161770:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc700 x1875444569504512/t0(0) o35->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572376.170966:0:161770:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569504512:192.168.202.157@tcp:35:lfs.0 00000100:00100000:1.0:1788572376.171030:0:161770:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572376.179657:0:161770:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:161770:1875444569504512:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572376.179695:0:161770:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cc000 x1875444569496064/t343597383686(343597383686) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572388 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572376.476204:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572376.987841:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569504768:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572376.987933:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572376.987941:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569504640:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572376.987973:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569504896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572376.994124:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569504896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572376.994141:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569504768:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572377.000248:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569504640:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572377.501710:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572378.524400:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572379.547941:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572380.572905:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:3.0:1788572381.206258:0:162187:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572381.206278:0:162187:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572381.206311:0:162187:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572381.206395:0:162187:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162187 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.206408:0:162187:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162187 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.206419:0:162187:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162187 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572381.206451:0:162187:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505024:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572381.206564:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abef00 going to sleep for 62 seconds 00000100:00100000:3.0:1788572381.217974:0:162187:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505024:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572381.218012:0:162187:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218027:0:162187:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218042:0:162187:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218055:0:162187:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218070:0:162187:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218078:0:162187:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218086:0:162187:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572381.218099:0:162187:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3f100 x1875444569505024/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572448 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572381.218121:0:162187:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218138:0:162187:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572381.218147:0:162187:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572381.218184:0:162187:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572381.218191:0:162187:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:3.0:1788572381.218197:0:162187:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572381.218201:0:162187:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7301fe7cb8 00010000:00010000:3.0:1788572381.218204:0:162187:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572381.218214:0:162187:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572381.219356:0:162187:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572381.219368:0:162187:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572381.219394:0:162187:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572381.219417:0:162187:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572381.219433:0:162187:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572381.220974:0:162187:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572381.220987:0:162187:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572381.220993:0:162187:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:3.0:1788572381.221007:0:162187:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7301fe7cb8 00010000:00010000:3.0:1788572381.221014:0:162187:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572381.221027:0:162187:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572381.231337:0:162187:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572381.231380:0:162187:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572381.231405:0:162187:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572381.231425:0:162187:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572381.231448:0:162187:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572381.231459:0:162187:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572381.231464:0:162187:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:2.0:1788572381.231474:0:162187:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7301fe7bb0 00010000:00010000:2.0:1788572381.231479:0:162187:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572381.231489:0:162187:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572381.231509:0:162187:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572381.231519:0:162187:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:2.0:1788572381.232069:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:2.0:1788572381.232093:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46ef7c00 uarg=0000000000000000 00800000:00000004:2.0:1788572381.232102:0:162187:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9089, max_age 9138 00000100:00100000:2.0:1788572381.232158:0:162187:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505152:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572381.232424:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896e40 going to sleep for 62 seconds 00000100:00100000:1.0:1788572381.239946:0:162187:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505152:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572381.239989:0:162187:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023635/1024000 00000080:00200080:1.0:1788572381.240094:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.240118:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00800000:00000004:1.0:1788572381.240127:0:162187:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9089, max_age 9138 00000100:00100000:1.0:1788572381.240176:0:162187:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505280:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572381.240279:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 62 seconds 00000100:00100000:1.0:1788572381.247958:0:162187:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505280:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572381.247996:0:162187:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00200080:1.0:1788572381.248087:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248108:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248117:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248121:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248125:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248129:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248133:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248136:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248140:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248143:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248147:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248150:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248153:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248156:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248160:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248163:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248167:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248171:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248174:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248178:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248181:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248184:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248187:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248190:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248194:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248197:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248200:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248203:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248207:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248210:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248213:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248216:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248220:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248223:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248226:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248229:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248232:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248235:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248239:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248242:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248245:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248248:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248252:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248255:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248258:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248261:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248265:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248268:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248271:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248274:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248278:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248281:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248284:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248287:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248290:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248294:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248297:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248300:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248303:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248306:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248310:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248313:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248316:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248319:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248323:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248327:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248331:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248334:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248337:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248340:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248343:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248346:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248350:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248353:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248356:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248359:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248363:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248365:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248369:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248372:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248376:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248379:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248382:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248385:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248389:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248392:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248396:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248400:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248403:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248407:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248410:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248415:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248418:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248421:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248425:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248429:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248433:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248442:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248447:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248451:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248455:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248458:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248462:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248465:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248470:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248474:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248478:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248481:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248485:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248489:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248493:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248497:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248501:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248504:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248507:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248511:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248515:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248518:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248522:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248526:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248530:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248533:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248538:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248541:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248545:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00800000:00000080:1.0:1788572381.248548:0:162187:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00000080:00200080:1.0:1788572381.248552:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.248558:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901800 uarg=0000000000000000 00020000:00000004:1.0:1788572381.248568:0:162187:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9089, max_age 9138 00000100:00080000:1.0:1788572381.248853:0:162187:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9074 00000100:00080000:1.0:1788572381.248869:0:162187:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572381.251071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a46524e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569505408:192.168.202.157@tcp:8:lfs.0 00000100:00100000:2.0:1788572381.262281:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036000 going to sleep for 11 seconds 00000080:00200080:1.0:1788572381.270500:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.270546:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00020000:00000004:1.0:1788572381.270559:0:162187:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9089, max_age 9138 00000100:00080000:1.0:1788572381.270600:0:162187:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9058 00000100:00080000:1.0:1788572381.270620:0:162187:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572381.270635:0:162187:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572381.270670:0:162187:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a528cd880 x1875444569505664/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572381.270715:0:162187:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a528cc000 x1875444569505792/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572381.270747:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572381.276382:0:162187:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48173cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572381.281993:0:162187:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162187:1875444569505792:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572381.282033:0:162187:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:1.0:1788572381.282187:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282210:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282219:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282224:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282229:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282233:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282237:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282242:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282246:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282250:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282254:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282257:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282261:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282265:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282269:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282273:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282276:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282279:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282282:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282286:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282291:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282295:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282299:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282303:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282389:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282395:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282400:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282404:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282408:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282411:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282415:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282418:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282422:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282426:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282430:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282433:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282448:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282453:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282457:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282460:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282464:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282468:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282472:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282475:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282480:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282484:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282682:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282689:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282693:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282696:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282700:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282703:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282706:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282709:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282712:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282715:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282719:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282722:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282737:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282741:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282745:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282747:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282751:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282754:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282757:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282760:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282764:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282766:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282770:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282774:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282778:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282782:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282785:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282788:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282792:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282794:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282797:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282800:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282803:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282806:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282810:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282814:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282818:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282821:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282825:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282828:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282831:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282858:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282862:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282865:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282868:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282870:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282873:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282876:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282880:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282885:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282889:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282892:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282896:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282900:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282903:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282907:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282911:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282914:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282918:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282921:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282925:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282928:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282933:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282937:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282941:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282945:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282948:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282952:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282956:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282959:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282963:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282967:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282971:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282975:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282980:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282983:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282987:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282990:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200080:1.0:1788572381.282995:0:162187:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe692c1fc0 00020000:00000080:1.0:1788572381.282999:0:162187:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47901000 uarg=0000000000000000 00000080:00200008:1.0:1788572381.283019:0:162187:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:2.0:1788572381.600828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572382.187314:0:2398:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350979 gen 1 00000100:00100000:1.0:1788572382.189684:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506048:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572382.189759:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506304:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572382.189777:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569505920:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572382.189792:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506176:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572382.189806:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506432:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572382.620979:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572383.648063:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572384.667897:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572385.691826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572386.715721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572387.294821:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506688:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.294928:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506944:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.294964:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:1.0:1788572387.294970:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569506560:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.295004:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.295033:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.303369:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569506560:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.303394:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.303409:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.303421:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506688:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572387.303431:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569506944:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572387.741824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572388.635673:0:162468:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 00010000:00010000:2.0:1788572388.766897:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572390.627392:0:162616:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00010000:00010000:2.0:1788572390.819050:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572392.413935:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569507200:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.414075:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507456:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.414182:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507712:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.414221:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572392.414228:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507328:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.414253:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507584:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.419290:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507328:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.419317:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507584:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.419330:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569507200:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.419343:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507456:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572392.419357:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569507712:192.168.202.157@tcp:400:kworker.0 00000080:00200008:2.0:1788572392.824792:0:162656:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572392.824815:0:162656:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572392.824847:0:162656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572392.824874:0:162656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572392.824892:0:162656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572392.824901:0:162656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572392.824911:0:162656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572392.824915:0:162656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:2.0:1788572392.824925:0:162656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d17cb8 00010000:00010000:2.0:1788572392.824930:0:162656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572392.824941:0:162656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788572392.827711:0:162656:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572392.827726:0:162656:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572392.827754:0:162656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572392.827776:0:162656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572392.827795:0:162656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572392.827804:0:162656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572392.827814:0:162656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572392.827818:0:162656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:3.0:1788572392.827827:0:162656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d17cb8 00010000:00010000:3.0:1788572392.827831:0:162656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572392.827843:0:162656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572392.828274:0:162656:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572392.828341:0:162656:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572392.828353:0:162656:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572392.828365:0:162656:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572392.828372:0:162656:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572392.828377:0:162656:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572392.828380:0:162656:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:3.0:1788572392.828385:0:162656:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d17bb0 00010000:00010000:3.0:1788572392.828389:0:162656:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572392.828397:0:162656:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:3.0:1788572392.828406:0:162656:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572392.828417:0:162656:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:3.0:1788572392.828535:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:3.0:1788572392.828554:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54093400 uarg=0000000000000000 00800000:00000004:3.0:1788572392.828563:0:162656:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9139, max_age 9150 00000100:00100000:3.0:1788572392.828736:0:162656:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569507840:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572392.830545:0:162656:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.835530:0:162656:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569507840:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572392.835571:0:162656:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023635/1024000 00000080:00200080:1.0:1788572392.835711:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.835733:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00800000:00000004:1.0:1788572392.835741:0:162656:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9139, max_age 9150 00000100:00100000:1.0:1788572392.835786:0:162656:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569507968:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572392.835868:0:162656:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.843910:0:162656:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569507968:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572392.843961:0:162656:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00200080:1.0:1788572392.844150:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844174:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844182:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844186:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844191:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844195:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844200:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844204:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844209:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844212:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844217:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844220:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844224:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844228:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844233:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844237:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844241:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844245:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844267:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844272:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844277:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844281:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844285:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844288:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844364:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844371:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844376:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844380:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844384:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844388:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844392:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844396:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844400:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844403:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844407:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844411:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844415:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844419:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844423:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844427:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844431:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844434:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844438:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844442:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844445:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844449:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844453:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844456:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844460:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844462:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844466:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844469:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844473:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844476:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844479:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844482:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844485:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844489:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844492:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844494:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844498:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844501:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844504:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844507:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844511:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844514:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844518:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844522:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844526:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844529:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844533:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844536:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844540:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844544:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844548:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844552:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844556:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844560:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844565:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844568:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844573:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844577:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844581:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844634:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844642:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844647:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844651:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844654:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844659:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844663:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844667:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844671:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844675:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844679:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844683:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844687:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844690:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844694:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844698:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844702:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844705:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844709:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844713:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844716:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844720:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844723:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844727:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844730:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844733:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844736:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844739:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844743:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844747:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844750:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844755:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844758:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844763:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844766:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844770:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844774:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844778:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844781:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844786:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844789:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844794:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00800000:00000080:1.0:1788572392.844798:0:162656:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.844802:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.844809:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00020000:00000004:1.0:1788572392.844819:0:162656:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9139, max_age 9150 00000100:00100000:1.0:1788572392.844873:0:162656:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569508096:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572392.844991:0:162656:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.850096:0:162656:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569508096:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572392.850133:0:162656:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:1.0:1788572392.850382:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.850406:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00020000:00000004:1.0:1788572392.850415:0:162656:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9139, max_age 9150 00000100:00100000:1.0:1788572392.850466:0:162656:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569508224:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572392.850570:0:162656:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572392.860002:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572392.863004:0:162656:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162656:1875444569508224:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572392.863055:0:162656:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:1.0:1788572392.863459:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863485:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863494:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863499:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863503:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863507:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863511:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863515:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863519:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863523:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863526:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863530:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863534:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863537:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863541:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863544:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863548:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863551:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863555:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863559:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863563:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863567:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863572:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863577:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863580:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863618:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863625:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863630:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863634:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863637:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863641:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863645:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863649:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863653:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863657:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863660:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863665:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863669:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863672:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863676:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863679:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863682:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863686:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863690:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863694:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863697:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863701:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863704:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863722:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863727:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863732:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863736:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863739:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863743:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863747:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863751:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863755:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863758:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863762:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863766:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863770:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863773:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863777:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863780:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863784:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863787:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863796:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863801:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863805:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863809:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863813:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863816:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863820:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863825:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863829:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863832:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863837:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863841:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863845:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863848:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863852:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863856:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863860:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863863:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863873:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863876:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863880:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863883:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863887:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863891:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863895:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863899:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863902:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863906:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863909:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863913:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863917:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863921:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863925:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863929:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863933:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863936:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863940:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863944:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863947:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863951:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863969:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863982:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863987:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863990:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.863994:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.863998:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864002:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864006:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864010:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864014:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864018:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864021:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864025:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864028:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864032:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864035:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864039:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864043:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200080:1.0:1788572392.864047:0:162656:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffee276ee0 00020000:00000080:1.0:1788572392.864051:0:162656:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47903c00 uarg=0000000000000000 00000080:00200008:1.0:1788572392.864069:0:162656:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:1.0:1788572392.941088:0:162673:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572392.941132:0:162673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572392.941321:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572392.941343:0:162673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00000080:00200000:1.0:1788572392.941355:0:162673:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572392.941366:0:162673:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572392.941370:0:162673:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572392.941381:0:162673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300dabbf8 00010000:00010000:1.0:1788572392.941386:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 1 00010000:00010000:1.0:1788572392.941397:0:162673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:1.0:1788572392.941416:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572392.941435:0:162673:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=27 00000080:00200000:1.0:1788572392.941441:0:162673:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:1.0:1788572392.941457:0:162673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572392.941519:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.941528:0:162673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.941538:0:162673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572392.941573:0:162673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508352:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572392.941761:0:162673:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.954916:0:162673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508352:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572392.957260:0:162673:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957329:0:162673:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957338:0:162673:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957352:0:162673:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957368:0:162673:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957392:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957406:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957414:0:162673:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572392.957435:0:162673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a528cc000 x1875444569508352/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572408 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572392.957453:0:162673:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.957465:0:162673:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/2 (0 0) 00000080:00200000:1.0:1788572392.957473:0:162673:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572392.957510:0:162673:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00200000:1.0:1788572392.958833:0:162673:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x2:0x0](ffff920a50d3cb08): rc = 0 00000080:00010000:1.0:1788572392.958851:0:162673:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb3ba 00000080:00010000:1.0:1788572392.958868:0:162673:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00010000:1.0:1788572392.958872:0:162673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3ba from it ffffaf7300dabac8 00010000:00010000:1.0:1788572392.958879:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.958892:0:162673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572392.958937:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572392.958967:0:162673:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), file ffff920a51751380, open_flags 18800, mode 0 opened 0 00000080:00200000:1.0:1788572392.958976:0:162673:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), intent=open 00000002:00010000:1.0:1788572392.959007:0:162673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x2:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788572392.959090:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.959099:0:162673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572392.959114:0:162673:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572392.959128:0:162673:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc000 x1875444569508608/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572392.959138:0:162673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572392.959166:0:162673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508608:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572392.959415:0:162673:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.985064:0:162673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508608:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572392.985458:0:162673:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985491:0:162673:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985501:0:162673:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985514:0:162673:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985533:0:162673:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985551:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985560:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985569:0:162673:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572392.985622:0:162673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=0 req@ffff920a528cc000 x1875444569508608/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572408 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572392.985639:0:162673:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.985652:0:162673:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 9663686441/1 (0 0) 00000100:00100000:1.0:1788572392.985816:0:162673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508736:192.168.202.157@tcp:900:lfs.0 00000100:00100000:1.0:1788572392.985920:0:162673:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572392.993693:0:162673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508736:192.168.202.157@tcp:900:lfs.0 00000002:00010000:1.0:1788572392.993762:0:162673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x2:0x0]) in obj [0x240002729:0x1:0x0], intent: open flags 0304001 00010000:00010000:1.0:1788572392.993850:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572392.993868:0:162673:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572392.993881:0:162673:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572392.993891:0:162673:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cfb80 x1875444569508992/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/131800 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572392.993900:0:162673:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572392.993923:0:162673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508992:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572392.994165:0:162673:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572393.000106:0:162673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569508992:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572393.000140:0:162673:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000158:0:162673:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000167:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000176:0:162673:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 3/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000187:0:162673:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000194:0:162673:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79600 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572393.000201:0:162673:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000210:0:162673:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 4/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000219:0:162673:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 5/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000228:0:162673:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572393.000237:0:162673:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 3/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000245:0:162673:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb3c8 lrc: 0/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162673 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572393.000290:0:162673:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a528cfb80 x1875444569508992/t21474837478(21474837478) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572408 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572393.000318:0:162673:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cfb80 x1875444569508992/t21474837478(21474837478) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572408 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572393.000426:0:162673:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572393.000466:0:162673:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000080:00200000:3.0:1788572393.000470:0:162673:0:(namei.c:147:ll_iget()) got inode: [0x240002729:0x1:0x0](ffff920a50d4db88): rc = 0 00000080:00010000:3.0:1788572393.000476:0:162673:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb3c1 00000080:00010000:3.0:1788572393.000495:0:162673:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00200008:3.0:1788572393.000515:0:162673:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 304000 00000002:00100000:3.0:1788572393.000544:0:162673:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cfb80 x1875444569508992/t21474837478(21474837478) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572408 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572393.000624:0:162673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3c1 from it ffff920a48138d80 00010000:00010000:3.0:1788572393.000630:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.000642:0:162673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572393.000667:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 44 00000080:00200000:3.0:1788572393.000756:0:162673:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572393.000778:0:162673:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572393.000798:0:162673:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572393.000818:0:162673:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572393.000826:0:162673:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572393.000834:0:162673:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572393.000838:0:162673:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:3.0:1788572393.000842:0:162673:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300dabbf8 00010000:00010000:3.0:1788572393.000845:0:162673:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572393.000853:0:162673:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572393.000863:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572393.000871:0:162673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=24 00000080:00200000:3.0:1788572393.000877:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572393.000881:0:162673:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=264 00000080:00200000:3.0:1788572393.000886:0:162673:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 44 00000080:00200008:3.0:1788572393.000890:0:162673:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 504000 00000080:00200008:3.0:1788572393.000916:0:162673:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 304000 00000080:00200080:3.0:1788572393.000973:0:162673:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) cmd=800466af arg=7ffc7b72cc3c 00000080:00200008:3.0:1788572393.001142:0:162673:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 504000 00000002:00100000:3.0:1788572393.001205:0:162673:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cfb80 x1875444569508992/t21474837478(21474837478) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572408 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572393.001218:0:162673:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572393.001224:0:162673:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3c380 x1875444569509248/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/568 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572393.001254:0:162673:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569509248:192.168.202.157@tcp:35:lfs.0 00000100:00100000:3.0:1788572393.001355:0:162673:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.006457:0:162673:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162673:1875444569509248:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572393.006516:0:162673:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cfb80 x1875444569508992/t21474837478(21474837478) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572408 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572393.079232:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525c00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569509504:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.079272:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079321:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079331:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079344:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079362:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079378:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079387:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079396:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.079411:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a46525c00 x1875444569509504/t21474837480(21474837480) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.079438:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46525c00 x1875444569509504/t21474837480(21474837480) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.079454:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.079469:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.079482:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/2 (0 0) 00000080:00200000:1.0:1788572393.079490:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.079507:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-0, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.079553:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x2:0x0](ffff920a50d4ec08) 00000080:00200000:1.0:1788572393.079702:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.079708:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x2:0x0](ffff920a50d4ec08) 00000080:00010000:1.0:1788572393.080253:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2:0x0](ffff920a50d4ec08) for lock 0x8364c713e25bb3cf 00000080:00010000:1.0:1788572393.080270:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2:0x0](ffff920a50d4ec08) 00000080:00200008:1.0:1788572393.080290:0:162678:0:(file.c:1090:ll_file_open()) START file f-0:[0x240002729:0x2:0x0](ffff920a50d4ec08), flags 100102 00000002:00100000:1.0:1788572393.080382:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46525c00 x1875444569509504/t21474837480(21474837480) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.080428:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3cf from it ffff920a481e5d20 00010000:00010000:1.0:1788572393.080435:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.080447:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.080473:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2:0x0](ffff920a50d4ec08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.080509:0:162678:0:(file.c:479:ll_file_release()) START file f-0:[0x240002729:0x2:0x0](ffff920a50d4ec08), flags 100002 00000002:00100000:1.0:1788572393.082845:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a46525c00 x1875444569509504/t21474837480(21474837480) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.082868:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.082874:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cfb80 x1875444569509760/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.082910:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569509760:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.083050:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.086515:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cfb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569509760:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.086569:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a46525c00 x1875444569509504/t21474837480(21474837480) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.086679:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.086703:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.086744:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572393.086763:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572393.086774:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.086782:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.086786:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.086796:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.086802:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572393.086812:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572393.086831:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.086840:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.086847:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.086850:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.086854:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.086868:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.086874:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-1, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d1680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.086884:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-1, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.086910:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-1,[0x240002729:0x3:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.086992:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.087002:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.087015:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.087023:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528ce680 x1875444569510016/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.087031:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.087055:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510016:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.087242:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.101995:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ce680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510016:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.102035:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.102087:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.102098:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.102111:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.102124:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.102140:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.105871:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.105887:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.105916:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528ce680 x1875444569510016/t21474837482(21474837482) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.105932:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528ce680 x1875444569510016/t21474837482(21474837482) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.105952:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.105987:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.106004:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/3 (0 0) 00000080:00200000:1.0:1788572393.106013:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.106032:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-1, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.106092:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3:0x0](ffff920a50d4d348) 00000080:00200000:1.0:1788572393.106212:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.106218:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3:0x0](ffff920a50d4d348) 00000080:00010000:1.0:1788572393.106223:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3:0x0](ffff920a50d4d348) for lock 0x8364c713e25bb3d6 00000080:00010000:1.0:1788572393.106237:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3:0x0](ffff920a50d4d348) 00000080:00200008:1.0:1788572393.106255:0:162678:0:(file.c:1090:ll_file_open()) START file f-1:[0x240002729:0x3:0x0](ffff920a50d4d348), flags 100102 00000002:00100000:1.0:1788572393.106281:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528ce680 x1875444569510016/t21474837482(21474837482) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.106336:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3d6 from it ffff920a48138a20 00010000:00010000:1.0:1788572393.106342:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.106354:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.106381:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3:0x0](ffff920a50d4d348), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.106415:0:162678:0:(file.c:479:ll_file_release()) START file f-1:[0x240002729:0x3:0x0](ffff920a50d4d348), flags 100002 00000002:00100000:1.0:1788572393.106472:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528ce680 x1875444569510016/t21474837482(21474837482) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.106485:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.106491:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cf480 x1875444569510272/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.106519:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510272:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.106665:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.113394:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510272:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.113448:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528ce680 x1875444569510016/t21474837482(21474837482) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.113538:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.113561:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.113628:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572393.113647:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572393.113657:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.113664:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.113668:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.113678:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.113683:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572393.113694:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572393.113826:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.113837:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.113844:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.113848:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.113852:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.113867:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.113872:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-2, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d0180, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.113882:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-2, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.113907:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-2,[0x240002729:0x4:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.113985:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.113994:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.114007:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.114015:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cf800 x1875444569510528/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.114025:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.114050:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510528:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.115126:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.137710:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46524380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569510784:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.137776:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cf800 x1875444569510528/t21474837484(21474837484) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.137905:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.137931:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.137956:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572393.137989:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572393.138003:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.138012:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.138017:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.138026:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.138030:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572393.138041:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572393.138061:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.138071:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.138079:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.138083:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.138087:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.138102:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.138107:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-3, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d1680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.138116:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-3, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.138142:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-3,[0x240002729:0x5:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.138214:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.138223:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.138238:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.138246:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cd180 x1875444569511040/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.138257:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.138636:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569511040:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.139678:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.171119:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569511040:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.171156:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171190:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171201:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171215:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171228:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171244:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171253:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171267:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.171281:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528cd180 x1875444569511040/t21474837486(21474837486) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.171311:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd180 x1875444569511040/t21474837486(21474837486) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.171329:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.171344:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171357:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/5 (0 0) 00000080:00200000:1.0:1788572393.171365:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.171383:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-3, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.171423:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x5:0x0](ffff920a50d4b248) 00000080:00200000:1.0:1788572393.171575:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x5:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.171581:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x5:0x0](ffff920a50d4b248) 00000080:00010000:1.0:1788572393.171625:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x5:0x0](ffff920a50d4b248) for lock 0x8364c713e25bb3e4 00000080:00010000:1.0:1788572393.171640:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x5:0x0](ffff920a50d4b248) 00000080:00200008:1.0:1788572393.171658:0:162678:0:(file.c:1090:ll_file_open()) START file f-3:[0x240002729:0x5:0x0](ffff920a50d4b248), flags 100102 00000002:00100000:1.0:1788572393.171682:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd180 x1875444569511040/t21474837486(21474837486) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.171721:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3e4 from it ffff920a48138d80 00010000:00010000:1.0:1788572393.171728:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.171738:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.171766:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x5:0x0](ffff920a50d4b248), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.171791:0:162678:0:(file.c:479:ll_file_release()) START file f-3:[0x240002729:0x5:0x0](ffff920a50d4b248), flags 100002 00000002:00100000:1.0:1788572393.171846:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cd180 x1875444569511040/t21474837486(21474837486) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.171858:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.171864:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528ce300 x1875444569511296/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.171890:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ce300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569511296:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.172020:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d000 going to sleep for 11 seconds 00000100:00100000:3.0:1788572393.186975:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46525500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569511552:192.168.202.157@tcp:101:createmany.0 00010000:00010000:3.0:1788572393.187014:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187066:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/0,1 mode: --/PR res: [0x240002729:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187077:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/0,1 mode: --/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187093:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187116:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187133:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187142:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187151:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572393.187170:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a46525500 x1875444569511552/t21474837488(21474837488) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572393.187183:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46525500 x1875444569511552/t21474837488(21474837488) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572393.187199:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572393.187213:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187225:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/6 (0 0) 00000080:00200000:3.0:1788572393.187233:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572393.187250:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-4, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:3.0:1788572393.187299:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x6:0x0](ffff920a50d3e3c8) 00000080:00200000:3.0:1788572393.187403:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x6:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572393.187409:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x6:0x0](ffff920a50d3e3c8) 00000080:00010000:3.0:1788572393.187413:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x6:0x0](ffff920a50d3e3c8) for lock 0x8364c713e25bb3eb 00000080:00010000:3.0:1788572393.187425:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x6:0x0](ffff920a50d3e3c8) 00000080:00200008:3.0:1788572393.187444:0:162678:0:(file.c:1090:ll_file_open()) START file f-4:[0x240002729:0x6:0x0](ffff920a50d3e3c8), flags 100102 00000002:00100000:3.0:1788572393.187470:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46525500 x1875444569511552/t21474837488(21474837488) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572393.187514:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3eb from it ffff920a481e5720 00010000:00010000:3.0:1788572393.187519:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.187534:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572393.187561:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x6:0x0](ffff920a50d3e3c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788572393.187632:0:162678:0:(file.c:479:ll_file_release()) START file f-4:[0x240002729:0x6:0x0](ffff920a50d3e3c8), flags 100002 00000002:00100000:3.0:1788572393.187689:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a46525500 x1875444569511552/t21474837488(21474837488) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572393.187700:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572393.187705:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3d180 x1875444569511808/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572393.187732:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569511808:192.168.202.157@tcp:35:createmany.0 00000100:00100000:3.0:1788572393.187818:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe900 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.225855:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569512320:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572393.225907:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a46526d80 x1875444569512064/t21474837490(21474837490) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572393.226024:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572393.226995:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572393.227028:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572393.227355:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:2.0:1788572393.227369:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572393.227381:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572393.227385:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:2.0:1788572393.227395:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:2.0:1788572393.227400:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:2.0:1788572393.227411:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:2.0:1788572393.227433:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.227445:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:2.0:1788572393.227455:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.227459:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572393.227462:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.227482:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.227487:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-6, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a46837680, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572393.227498:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-6, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:2.0:1788572393.227534:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-6,[0x240002729:0x8:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572393.227634:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb3f9 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.227645:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb3f9 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572393.227659:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572393.227672:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890000 x1875444569512576/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572393.227683:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb3f9 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572393.227710:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569512576:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572393.227923:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.302373:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5cc0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.312634:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46527800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569513856:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.312701:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a46527100 x1875444569513600/t21474837496(21474837496) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.312831:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.312859:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.312886:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788572393.312904:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:1.0:1788572393.312914:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.312921:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.312925:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.312936:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.312941:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:1.0:1788572393.312951:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788572393.312981:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.312991:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.312999:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.313004:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.313007:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.313028:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.313034:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-9, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d0600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.313044:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-9, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.313084:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-9,[0x240002729:0xb:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.315361:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.315385:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.315402:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.315418:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc380 x1875444569514112/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.315429:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.315919:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514112:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.317448:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8de40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.339825:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514112:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.339865:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339914:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/0,1 mode: --/PR res: [0x240002729:0xb:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339922:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/0,1 mode: --/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339934:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339949:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339978:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.339987:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.340001:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 4/1,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.340015:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528cc380 x1875444569514112/t21474837498(21474837498) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.340030:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cc380 x1875444569514112/t21474837498(21474837498) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.340046:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.340366:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.340379:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/11 (0 0) 00000080:00200000:1.0:1788572393.340387:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.340405:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-9, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.341494:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0xb:0x0](ffff920a50d4f448) 00000080:00200000:1.0:1788572393.341650:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0xb:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.341659:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0xb:0x0](ffff920a50d4f448) 00000080:00010000:1.0:1788572393.341663:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0xb:0x0](ffff920a50d4f448) for lock 0x8364c713e25bb40e 00000080:00010000:1.0:1788572393.341678:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0xb:0x0](ffff920a50d4f448) 00000080:00200008:1.0:1788572393.341698:0:162678:0:(file.c:1090:ll_file_open()) START file f-9:[0x240002729:0xb:0x0](ffff920a50d4f448), flags 100102 00000002:00100000:1.0:1788572393.341739:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cc380 x1875444569514112/t21474837498(21474837498) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.341790:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb40e from it ffff920a48138d80 00010000:00010000:1.0:1788572393.341799:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.341813:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.341842:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0xb:0x0](ffff920a50d4f448), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.341884:0:162678:0:(file.c:479:ll_file_release()) START file f-9:[0x240002729:0xb:0x0](ffff920a50d4f448), flags 100002 00000002:00100000:1.0:1788572393.341945:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cc380 x1875444569514112/t21474837498(21474837498) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.341956:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.341973:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cea00 x1875444569514368/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.342002:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514368:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.342146:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.361681:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cea00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514368:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.361731:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cc380 x1875444569514112/t21474837498(21474837498) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.361834:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.361858:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.361882:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572393.361900:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:1.0:1788572393.361915:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.361922:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.361926:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.361936:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.361941:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:1.0:1788572393.361950:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572393.361980:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.361989:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.361995:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.361999:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.362002:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.362020:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.362025:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-10, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d0900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.362035:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-10, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.362070:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-10,[0x240002729:0xc:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.362147:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.362156:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.362169:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.362176:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cce00 x1875444569514624/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.362187:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.362210:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514624:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.362400:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.373339:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cce00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514624:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.373381:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373424:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/0,1 mode: --/PR res: [0x240002729:0xc:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373446:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/0,1 mode: --/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373462:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373474:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373491:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373501:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373515:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.373526:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528cce00 x1875444569514624/t21474837500(21474837500) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.373541:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cce00 x1875444569514624/t21474837500(21474837500) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.373559:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.373574:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373655:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/12 (0 0) 00000080:00200000:1.0:1788572393.373665:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.373682:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-10, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.373725:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0xc:0x0](ffff920a50d480c8) 00000080:00200000:1.0:1788572393.373832:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0xc:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.373838:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0xc:0x0](ffff920a50d480c8) 00000080:00010000:1.0:1788572393.373842:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0xc:0x0](ffff920a50d480c8) for lock 0x8364c713e25bb415 00000080:00010000:1.0:1788572393.373857:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0xc:0x0](ffff920a50d480c8) 00000080:00200008:1.0:1788572393.373875:0:162678:0:(file.c:1090:ll_file_open()) START file f-10:[0x240002729:0xc:0x0](ffff920a50d480c8), flags 100102 00000002:00100000:1.0:1788572393.373901:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cce00 x1875444569514624/t21474837500(21474837500) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.373947:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb415 from it ffff920a48138d80 00010000:00010000:1.0:1788572393.373953:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.373973:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.374002:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0xc:0x0](ffff920a50d480c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.374044:0:162678:0:(file.c:479:ll_file_release()) START file f-10:[0x240002729:0xc:0x0](ffff920a50d480c8), flags 100002 00000002:00100000:1.0:1788572393.374102:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cce00 x1875444569514624/t21474837500(21474837500) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.374113:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.374118:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cca80 x1875444569514880/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.374144:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cca80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514880:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.374252:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.380446:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cca80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569514880:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.380496:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cce00 x1875444569514624/t21474837500(21474837500) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.380658:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.380683:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.380707:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572393.380725:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572393.380742:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.380749:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.380754:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.380763:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.380767:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572393.380778:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572393.380796:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.380804:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.380811:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.380815:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.380818:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.380832:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.380837:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-11, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d0600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.380846:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-11, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.380871:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-11,[0x240002729:0xd:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.380938:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.380946:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.380960:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.380978:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cd500 x1875444569515136/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.380990:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.381015:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569515136:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.381223:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d900 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.387717:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569515136:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.387757:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387798:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/0,1 mode: --/PR res: [0x240002729:0xd:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387807:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/0,1 mode: --/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387821:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387833:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387850:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387859:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387871:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.387886:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528cd500 x1875444569515136/t21474837502(21474837502) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.387900:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd500 x1875444569515136/t21474837502(21474837502) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.387916:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.387930:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.387942:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/13 (0 0) 00000080:00200000:1.0:1788572393.387951:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.387981:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-11, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.388025:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0xd:0x0](ffff920a50d4e3c8) 00000080:00200000:1.0:1788572393.388143:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0xd:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.388150:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0xd:0x0](ffff920a50d4e3c8) 00000080:00010000:1.0:1788572393.388153:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0xd:0x0](ffff920a50d4e3c8) for lock 0x8364c713e25bb41c 00000080:00010000:1.0:1788572393.388167:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0xd:0x0](ffff920a50d4e3c8) 00000080:00200008:1.0:1788572393.388189:0:162678:0:(file.c:1090:ll_file_open()) START file f-11:[0x240002729:0xd:0x0](ffff920a50d4e3c8), flags 100102 00000002:00100000:1.0:1788572393.388220:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd500 x1875444569515136/t21474837502(21474837502) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.388274:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb41c from it ffff920a48138d80 00010000:00010000:1.0:1788572393.388279:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.388290:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.388320:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0xd:0x0](ffff920a50d4e3c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.388360:0:162678:0:(file.c:479:ll_file_release()) START file f-11:[0x240002729:0xd:0x0](ffff920a50d4e3c8), flags 100002 00000002:00100000:1.0:1788572393.388419:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cd500 x1875444569515136/t21474837502(21474837502) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.388431:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.391363:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc000 x1875444569515392/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.391416:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569515392:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.391518:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.409914:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a46526300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569515904:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.409983:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a46526a00 x1875444569515648/t21474837504(21474837504) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.410115:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.410139:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.410165:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572393.410183:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572393.410200:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.410207:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.410211:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.410220:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.410224:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572393.410235:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572393.410265:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.410275:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.410283:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.410287:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.410291:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.410305:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.410310:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-13, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d1b00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.410320:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-13, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.410346:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-13,[0x240002729:0xf:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.410415:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.410425:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.410449:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.410458:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cd880 x1875444569516160/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.410468:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.410494:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516160:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.410698:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.417798:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cd880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516160:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.417837:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417881:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/0,1 mode: --/PR res: [0x240002729:0xf:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417891:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/0,1 mode: --/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417905:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417916:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/0,1 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417931:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417940:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 5/1,1 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.417953:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 4/1,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.417979:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528cd880 x1875444569516160/t21474837506(21474837506) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.417994:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd880 x1875444569516160/t21474837506(21474837506) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.418010:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.418025:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.418039:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/15 (0 0) 00000080:00200000:1.0:1788572393.418047:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.418064:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-13, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:1.0:1788572393.418105:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0xf:0x0](ffff920a50d4cb08) 00000080:00200000:1.0:1788572393.418215:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0xf:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572393.418222:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0xf:0x0](ffff920a50d4cb08) 00000080:00010000:1.0:1788572393.418226:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0xf:0x0](ffff920a50d4cb08) for lock 0x8364c713e25bb42a 00000080:00010000:1.0:1788572393.418238:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0xf:0x0](ffff920a50d4cb08) 00000080:00200008:1.0:1788572393.418257:0:162678:0:(file.c:1090:ll_file_open()) START file f-13:[0x240002729:0xf:0x0](ffff920a50d4cb08), flags 100102 00000002:00100000:1.0:1788572393.418284:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cd880 x1875444569516160/t21474837506(21474837506) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572393.418332:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb42a from it ffff920a48138d80 00010000:00010000:1.0:1788572393.418338:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.418348:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.418377:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0xf:0x0](ffff920a50d4cb08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572393.418417:0:162678:0:(file.c:479:ll_file_release()) START file f-13:[0x240002729:0xf:0x0](ffff920a50d4cb08), flags 100002 00000002:00100000:1.0:1788572393.418484:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cd880 x1875444569516160/t21474837506(21474837506) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.418496:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.418502:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cc700 x1875444569516416/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.418531:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516416:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572393.418660:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.424178:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cc700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516416:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572393.424232:0:162678:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cd880 x1875444569516160/t21474837506(21474837506) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.424370:0:162678:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.424400:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.424428:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572393.424460:0:162678:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572393.424475:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.424483:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.424487:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.424497:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf7300d9bbf8 00010000:00010000:1.0:1788572393.424501:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572393.424512:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788572393.424531:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.424539:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.424546:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.424550:0:162678:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.424553:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.424578:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.424583:0:162678:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-14, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), file ffff920a546d0900, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572393.424645:0:162678:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-14, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000002:00010000:1.0:1788572393.424671:0:162678:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-14,[0x240002729:0x10:0x0]) in obj [0x240002729:0x1:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572393.424740:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.424749:0:162678:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.424762:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.424770:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528ced80 x1875444569516672/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.424780:0:162678:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162678 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.424805:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516672:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572393.424995:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.448637:0:162678:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528ced80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516672:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572393.448678:0:162678:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448868:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x10:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448880:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448894:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448908:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448926:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448936:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.448951:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.448976:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a528ced80 x1875444569516672/t21474837508(21474837508) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.448991:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528ced80 x1875444569516672/t21474837508(21474837508) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.449008:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572393.454351:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.454397:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/16 (0 0) 00000080:00200000:2.0:1788572393.454407:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572393.454429:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-14, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:2.0:1788572393.454485:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x10:0x0](ffff920a50d29148) 00000080:00200000:2.0:1788572393.454664:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x10:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572393.454671:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x10:0x0](ffff920a50d29148) 00000080:00010000:2.0:1788572393.454676:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x10:0x0](ffff920a50d29148) for lock 0x8364c713e25bb431 00000080:00010000:2.0:1788572393.454690:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x10:0x0](ffff920a50d29148) 00000080:00200008:2.0:1788572393.454710:0:162678:0:(file.c:1090:ll_file_open()) START file f-14:[0x240002729:0x10:0x0](ffff920a50d29148), flags 100102 00000002:00100000:2.0:1788572393.454994:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528ced80 x1875444569516672/t21474837508(21474837508) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572393.455044:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb431 from it ffff920a48138d80 00010000:00010000:2.0:1788572393.455051:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.455063:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.455094:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x10:0x0](ffff920a50d29148), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572393.455247:0:162678:0:(file.c:479:ll_file_release()) START file f-14:[0x240002729:0x10:0x0](ffff920a50d29148), flags 100002 00000002:00100000:2.0:1788572393.455403:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528ced80 x1875444569516672/t21474837508(21474837508) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572393.455418:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572393.455425:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890380 x1875444569516928/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572393.455458:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569516928:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572393.456472:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036d80 going to sleep for 11 seconds 00000002:00100000:2.0:1788572393.485295:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a46527480 x1875444569517696/t21474837512(21474837512) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572393.485351:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb43f from it ffff920a52f77d20 00010000:00010000:2.0:1788572393.485359:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb43f lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be69c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.485371:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb43f lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be69c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.485407:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x12:0x0](ffff920a50d41148), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572393.485449:0:162678:0:(file.c:479:ll_file_release()) START file f-16:[0x240002729:0x12:0x0](ffff920a50d41148), flags 100002 00000002:00100000:2.0:1788572393.485509:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a46527480 x1875444569517696/t21474837512(21474837512) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572393.485522:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572393.485528:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890a80 x1875444569517952/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572393.485555:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569517952:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572393.485998:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036b40 going to sleep for 11 seconds 00010000:00010000:2.0:1788572393.548929:0:162678:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x15:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.548950:0:162678:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.548964:0:162678:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.548987:0:162678:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.549007:0:162678:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.549017:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.549025:0:162678:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572393.549042:0:162678:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a478bed80 x1875444569519232/t21474837518(21474837518) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572393.549056:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a478bed80 x1875444569519232/t21474837518(21474837518) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572393.549071:0:162678:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572393.549087:0:162678:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.549276:0:162678:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/21 (0 0) 00000080:00200000:2.0:1788572393.549286:0:162678:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572393.549306:0:162678:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-19, dir=[0x240002729:0x1:0x0](ffff920a50d4db88), intent=open|creat 00000080:00200000:2.0:1788572393.549353:0:162678:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x15:0x0](ffff920a50d2ec08) 00000080:00200000:2.0:1788572393.549453:0:162678:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x15:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572393.549460:0:162678:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x15:0x0](ffff920a50d2ec08) 00000080:00010000:2.0:1788572393.549464:0:162678:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x15:0x0](ffff920a50d2ec08) for lock 0x8364c713e25bb454 00000080:00010000:2.0:1788572393.549477:0:162678:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x15:0x0](ffff920a50d2ec08) 00000080:00200008:2.0:1788572393.549495:0:162678:0:(file.c:1090:ll_file_open()) START file f-19:[0x240002729:0x15:0x0](ffff920a50d2ec08), flags 100102 00000002:00100000:2.0:1788572393.549919:0:162678:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a478bed80 x1875444569519232/t21474837518(21474837518) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572393.549983:0:162678:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb454 from it ffff920a52f77f60 00010000:00010000:2.0:1788572393.549991:0:162678:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.550003:0:162678:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.550032:0:162678:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x15:0x0](ffff920a50d2ec08), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572393.550074:0:162678:0:(file.c:479:ll_file_release()) START file f-19:[0x240002729:0x15:0x0](ffff920a50d2ec08), flags 100002 00000002:00100000:2.0:1788572393.550152:0:162678:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a478bed80 x1875444569519232/t21474837518(21474837518) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572393.550173:0:162678:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572393.550179:0:162678:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52893800 x1875444569519488/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572393.550207:0:162678:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162678:1875444569519488:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572393.550323:0:162678:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036a80 going to sleep for 11 seconds 00000080:00200000:1.0:1788572393.574354:0:162684:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.574396:0:162684:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.574420:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788572393.574440:0:162684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:1.0:1788572393.574449:0:162684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.574461:0:162684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.574465:0:162684:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.574475:0:162684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf730153bbf8 00010000:00010000:1.0:1788572393.574479:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:1.0:1788572393.574495:0:162684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788572393.574515:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.574525:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=27 00000080:00200000:1.0:1788572393.575000:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.575007:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=27 00000080:00200000:1.0:1788572393.575011:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.575017:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=267 00000080:00200000:1.0:1788572393.575086:0:162684:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.575102:0:162684:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.575112:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572393.575123:0:162684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:1.0:1788572393.575131:0:162684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.575136:0:162684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.575139:0:162684:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.575143:0:162684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf730153bbf8 00010000:00010000:1.0:1788572393.575146:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:1.0:1788572393.575262:0:162684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788572393.575282:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.575287:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=27 00000080:00200000:1.0:1788572393.575290:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.575293:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=267 00000080:00200000:1.0:1788572393.575297:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572393.575308:0:162684:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 304000 00000002:00010000:1.0:1788572393.575330:0:162684:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: open flags 0100240304001 00010000:00010000:1.0:1788572393.575403:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.575412:0:162684:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.575429:0:162684:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.575447:0:162684:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cdc00 x1875444569519744/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.575458:0:162684:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.575486:0:162684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569519744:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572393.576123:0:162684:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.579178:0:162684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cdc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569519744:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572393.579210:0:162684:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579227:0:162684:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579241:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579249:0:162684:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579261:0:162684:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579269:0:162684:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79c00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572393.579274:0:162684:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579283:0:162684:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579291:0:162684:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 5/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579299:0:162684:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572393.579308:0:162684:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.579316:0:162684:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb45b lrc: 0/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.579347:0:162684:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a528cdc00 x1875444569519744/t21474837520(21474837520) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.579362:0:162684:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cdc00 x1875444569519744/t21474837520(21474837520) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.579382:0:162684:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572393.579419:0:162684:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000002:00100000:1.0:1788572393.579442:0:162684:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cdc00 x1875444569519744/t21474837520(21474837520) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.579575:0:162684:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.580418:0:162684:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.580447:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788572393.580466:0:162684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:1.0:1788572393.580475:0:162684:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.580484:0:162684:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.580489:0:162684:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.580499:0:162684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf730153bbf8 00010000:00010000:1.0:1788572393.580503:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:1.0:1788572393.580511:0:162684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788572393.580531:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.580542:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=25 00000080:00200000:1.0:1788572393.580550:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.580554:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572393.580558:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.580563:0:162684:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=265 00000080:00200000:1.0:1788572393.580569:0:162684:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3:0x0](ffff920a50d4d348), inode mode 81a4 mask 44 00000080:00200008:1.0:1788572393.580578:0:162684:0:(file.c:1090:ll_file_open()) START file f-1:[0x240002729:0x3:0x0](ffff920a50d4d348), flags 100400 00000002:00010000:1.0:1788572393.580679:0:162684:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x3:0x0]) in obj [0x240002729:0x1:0x0], intent: open flags 0240100401 00010000:00010000:1.0:1788572393.581153:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.581227:0:162684:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.581242:0:162684:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.581257:0:162684:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cdf80 x1875444569520000/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.581266:0:162684:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162684 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.581294:0:162684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cdf80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520000:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572393.581720:0:162684:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d3c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.589713:0:162684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cdf80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520000:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572393.589760:0:162684:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589795:0:162684:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589812:0:162684:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589831:0:162684:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589844:0:162684:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589861:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589871:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589880:0:162684:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.589894:0:162684:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a528cdf80 x1875444569520000/t21474837521(21474837521) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.589908:0:162684:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cdf80 x1875444569520000/t21474837521(21474837521) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.589923:0:162684:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572393.589938:0:162684:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.589952:0:162684:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.590136:0:162684:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.590147:0:162684:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.590163:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.590170:0:162684:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.590250:0:162684:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1788572393.590259:0:162684:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x240002729:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572393.590333:0:162684:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788572393.590339:0:162684:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3:0x0](ffff920a50d4d348) for lock 0x8364c713e25bb3d6 00000080:00010000:1.0:1788572393.590347:0:162684:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3d6 from it ffffaf730153bbf0 00010000:00010000:1.0:1788572393.590351:0:162684:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.590360:0:162684:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 1 00000002:00100000:1.0:1788572393.590394:0:162684:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a528cdf80 x1875444569520000/t21474837521(21474837521) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572393.590502:0:162684:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x240002729:0x3:0x0](ffff920a50d4d348) cmd=800466af arg=7ffd3a8fa6bc 00000080:00200008:1.0:1788572393.590537:0:162684:0:(file.c:479:ll_file_release()) START file f-1:[0x240002729:0x3:0x0](ffff920a50d4d348), flags 100000 00000002:00100000:1.0:1788572393.590658:0:162684:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cdf80 x1875444569520000/t21474837521(21474837521) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.590670:0:162684:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.590676:0:162684:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a528cf100 x1875444569520256/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.590703:0:162684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a528cf100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520256:192.168.202.157@tcp:35:lfs.0 00000100:00100000:1.0:1788572393.590832:0:162684:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00010000:00010000:1.0:1788572393.593619:0:160929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572393.593686:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 2/0,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.593694:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.595276:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.595296:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb462 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be728 expref: -99 pid: 162684 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.595323:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572393.595325:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572393.595328:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572393.600029:0:162684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a528cf100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520256:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572393.600080:0:162684:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cdf80 x1875444569520000/t21474837521(21474837521) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572393.600341:0:162684:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 304000 00000002:00100000:1.0:1788572393.600406:0:162684:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a528cdc00 x1875444569519744/t21474837520(21474837520) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.600419:0:162684:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.600425:0:162684:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353480 x1875444569520640/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.600451:0:162684:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520640:192.168.202.157@tcp:35:lfs.0 00000100:00100000:1.0:1788572393.600537:0:162684:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.604554:0:162684:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162684:1875444569520640:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572393.604640:0:162684:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a528cdc00 x1875444569519744/t21474837520(21474837520) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572393.666935:0:162685:0:(file.c:479:ll_file_release()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), flags 704000 00000002:00100000:2.0:1788572393.667068:0:162685:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a478bf100 x1875444569520896/t347892350981(347892350981) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572393.667085:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572393.667091:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52893b80 x1875444569521280/t0(0) o35->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572393.667279:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521280:192.168.202.157@tcp:35:rm.0 00000100:00100000:2.0:1788572393.667404:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.674238:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521280:192.168.202.157@tcp:35:rm.0 00000002:00100000:2.0:1788572393.674289:0:162685:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a478bf100 x1875444569520896/t347892350981(347892350981) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572393.674408:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572393.674430:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572393.674454:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572393.674471:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:2.0:1788572393.674479:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572393.674486:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572393.674490:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:2.0:1788572393.674500:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf73018dbbf8 00010000:00010000:2.0:1788572393.674505:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:2.0:1788572393.674514:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:2.0:1788572393.674533:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.674541:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80a.replay-single, flags=266 00000080:00200000:2.0:1788572393.674550:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572393.674561:0:162685:0:(file.c:1090:ll_file_open()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), flags 704400 00000002:00010000:2.0:1788572393.674579:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x2:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1788572393.674705:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.674713:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572393.674727:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572393.674735:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52893b80 x1875444569521536/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572393.674745:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572393.674768:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521536:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572393.677582:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.686544:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521536:192.168.202.157@tcp:101:rm.0 00010000:00010000:2.0:1788572393.686601:0:162685:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.686613:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.686626:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 4/1,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.686637:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 4/1,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572393.686651:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff920a52893b80 x1875444569521536/t347892350983(347892350983) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572393.686664:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a52893b80 x1875444569521536/t347892350983(347892350983) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572393.686685:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 3/1,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572393.686695:0:162685:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 16 res 8590095617/2 (0 0) 00000080:00200000:2.0:1788572393.686702:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572393.687011:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00010000:3.0:1788572393.687030:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb470 00000080:00010000:3.0:1788572393.687039:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb470 from it ffffaf73018dbbd0 00010000:00010000:3.0:1788572393.687043:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 3/1,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.687053:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:3.0:1788572393.687083:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a52893b80 x1875444569521536/t347892350983(347892350983) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:3.0:1788572393.687166:0:162685:0:(file.c:6296:ll_getattr_dentry()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572393.687175:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08),name=d80a.replay-single 00000002:00010000:3.0:1788572393.687195:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x2:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572393.687206:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572393.687218:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572393.687226:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572393.687233:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00010000:3.0:1788572393.687236:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb3ba 00000080:00010000:3.0:1788572393.687240:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3ba from it ffffaf73018dbcd8 00010000:00010000:3.0:1788572393.687243:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572393.687250:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572393.687549:0:162685:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) pos/size0/4096 32bit_api 0 00000080:00200000:3.0:1788572393.687574:0:162685:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:3.0:1788572393.687634:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572393.687643:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=0 00000080:00200008:3.0:1788572393.687654:0:162685:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572393.703547:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88),name=remote_dir 00000002:00010000:1.0:1788572393.703626:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.703723:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.703736:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.703748:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.703783:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521664:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.703920:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.711946:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521664:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.711987:0:162685:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712028:0:162685:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712037:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712049:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712061:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712074:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712082:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712089:0:162685:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.712101:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48353480 x1875444569521664/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.712116:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712132:0:162685:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/1 (0 0) 00000080:00200000:1.0:1788572393.712138:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572393.712292:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572393.712300:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb477 00000080:00010000:1.0:1788572393.712306:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00010000:1.0:1788572393.712420:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb477 from it ffffaf73018dbcb8 00010000:00010000:1.0:1788572393.712426:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712437:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.712510:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.712519:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:1.0:1788572393.712703:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572393.712713:0:162685:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 704400 00000002:00010000:1.0:1788572393.712737:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: open flags 0100240704401 00010000:00010000:1.0:1788572393.712787:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.712795:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.712806:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.712814:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353480 x1875444569521920/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.712824:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.712844:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521920:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.713146:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.724205:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569521920:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.724231:0:162685:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724246:0:162685:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724253:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724260:0:162685:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724268:0:162685:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724274:0:162685:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79c00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572393.724278:0:162685:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724285:0:162685:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 4/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724291:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 5/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724299:0:162685:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572393.724307:0:162685:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 3/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.724313:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb47e lrc: 0/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.724342:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a48353480 x1875444569521920/t21474837524(21474837524) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.724356:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48353480 x1875444569521920/t21474837524(21474837524) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.724372:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572393.724404:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000002:00100000:1.0:1788572393.724425:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48353480 x1875444569521920/t21474837524(21474837524) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572393.724491:0:162685:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572393.724498:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88),name=remote_dir 00000002:00010000:1.0:1788572393.724519:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.724533:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.724545:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.724552:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.724557:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00010000:1.0:1788572393.724560:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb477 00000080:00010000:1.0:1788572393.724565:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb477 from it ffffaf73018dbcd8 00010000:00010000:1.0:1788572393.724570:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.724580:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572393.724631:0:162685:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) pos/size0/4096 32bit_api 0 00000002:00010000:1.0:1788572393.724644:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x240002729:0x1:0x0], intent: readdir flags 0 00010000:00010000:1.0:1788572393.724651:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.724659:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1788572393.725386:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522048:192.168.202.157@tcp:37:rm.0 00000100:00100000:1.0:1788572393.725486:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.742138:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522048:192.168.202.157@tcp:37:rm.0 00010000:00010000:1.0:1788572393.742896:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.742913:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200008:1.0:1788572393.742986:0:162685:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 704000 00000002:00100000:1.0:1788572393.743056:0:162685:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48353480 x1875444569521920/t21474837524(21474837524) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.743068:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.743075:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350000 x1875444569522304/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.743097:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522304:192.168.202.157@tcp:35:rm.0 00000100:00100000:1.0:1788572393.743177:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.746808:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522304:192.168.202.157@tcp:35:rm.0 00000002:00100000:1.0:1788572393.746848:0:162685:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48353480 x1875444569521920/t21474837524(21474837524) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572393.746922:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.746932:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:1.0:1788572393.746940:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 44 00000080:00200008:1.0:1788572393.746948:0:162685:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 704400 00000002:00010000:1.0:1788572393.746982:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: open flags 0104240704401 00010000:00010000:1.0:1788572393.747047:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.747056:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572393.747067:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572393.747074:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350000 x1875444569522560/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572393.747083:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.747103:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522560:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.747353:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.750197:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522560:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.750242:0:162685:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.750254:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.750267:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 4/1,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.750279:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 4/1,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.750296:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff920a48350000 x1875444569522560/t21474837526(21474837526) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572393.750308:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569522560/t21474837526(21474837526) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.750323:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 3/1,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.750332:0:162685:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 16 res 9663686441/1 (0 0) 00000080:00200000:1.0:1788572393.750340:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572393.750380:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000080:00010000:1.0:1788572393.750388:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb485 00000080:00010000:1.0:1788572393.750396:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb485 from it ffffaf73018dbbd0 00010000:00010000:1.0:1788572393.750400:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 3/1,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.750415:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:1.0:1788572393.750443:0:162685:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569522560/t21474837526(21474837526) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572393.750510:0:162685:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572393.750516:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88),name=remote_dir 00000002:00010000:1.0:1788572393.750788:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x1:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.750800:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572393.750811:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00000080:00200000:1.0:1788572393.750818:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.750823:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00010000:1.0:1788572393.750826:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb477 00000080:00010000:1.0:1788572393.750830:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb477 from it ffffaf73018dbcd8 00010000:00010000:1.0:1788572393.750833:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 1 00010000:00010000:1.0:1788572393.750841:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000080:00200000:1.0:1788572393.750870:0:162685:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff920a47b93800) remote_dir proj=0 00000080:00000004:1.0:1788572393.750874:0:162685:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff920a47b93800: age 9058, max_age 9151 00800000:00000004:1.0:1788572393.750883:0:162685:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9151, max_age 9151 00800000:00000004:1.0:1788572393.750886:0:162685:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: use ffff920a46419e18 cache blocks 317230/353529 objects 1023635/1024000 00800000:00000004:1.0:1788572393.750889:0:162685:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9151, max_age 9151 00800000:00000004:1.0:1788572393.750891:0:162685:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: use ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00000004:1.0:1788572393.750896:0:162685:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff920a47b93800: update ffff920a4641b7d8 cache blocks 634557/707058 objects 2047358/2048000 00000080:00000004:1.0:1788572393.750899:0:162685:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 634557/707058 objects 2047358/2048000 00000080:00000004:1.0:1788572393.750902:0:162685:0:(obd_class.h:1090:obd_statfs()) lustre-clilov-ffff920a47b93800: age 8058, max_age 9151 00020000:00000004:1.0:1788572393.750916:0:162685:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0000-osc-ffff920a47b93800: age 9151, max_age 9151 00000008:00000004:1.0:1788572393.750920:0:162685:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0000-osc-ffff920a47b93800: use ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00020000:00000004:1.0:1788572393.750924:0:162685:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0001-osc-ffff920a47b93800: age 9151, max_age 9151 00000008:00000004:1.0:1788572393.750926:0:162685:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0001-osc-ffff920a47b93800: use ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00000004:1.0:1788572393.750934:0:162685:0:(obd_class.h:1116:obd_statfs()) lustre-clilov-ffff920a47b93800: update ffff920a4641d198 cache blocks 1802716/1916558 objects 523438/524288 00000080:00000004:1.0:1788572393.750937:0:162685:0:(llite_lib.c:2766:ll_statfs_internal()) OSC blocks 1802716/1916558 objects 523438/524288 00000080:00200000:1.0:1788572393.750953:0:162685:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) pos/size0/4096 32bit_api 0 00000080:00200000:1.0:1788572393.751214:0:162685:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:1.0:1788572393.751240:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.751251:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-13, flags=0 00000080:00200000:1.0:1788572393.751257:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.751261:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-13, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.751290:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572393.751425:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.751438:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xf:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.751473:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 1/0,0 mode: --/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.751486:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00010000:00010000:1.0:1788572393.751491:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788572393.751494:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb42a lrc: 0/0,0 mode: --/PR res: [0x240002729:0xf:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be633 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.751508:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb477 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be760 expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 1 00000020:00100000:1.0:1788572393.751522:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.751727:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353480 x1875444569522816/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.751751:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522816:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.751947:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.758743:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569522816:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.758812:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0xf:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.758825:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.758880:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0xf:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.758888:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0xf:0x0](ffff920a50d4cb08) 00000080:00200000:1.0:1788572393.762798:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.762821:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-2, flags=0 00000080:00200000:1.0:1788572393.762829:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.762835:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-2, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.762890:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be4b2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.762911:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x4:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.762964:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 1/0,0 mode: --/PR res: [0x240002729:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4b2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.762984:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.762987:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb3dd lrc: 0/0,0 mode: --/PR res: [0x240002729:0x4:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4b2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572393.763620:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.763738:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351500 x1875444569523072/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.763767:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523072:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.763909:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.770101:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523072:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.770178:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x4:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.770191:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.770218:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x4:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.770225:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x4:0x0](ffff920a50d43248) 00000080:00200000:1.0:1788572393.770306:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.770329:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-7, flags=0 00000080:00200000:1.0:1788572393.770335:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.770340:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-7, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.770382:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb400 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be561 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.770401:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.770447:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb400 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be561 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.770454:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.770457:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb400 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be561 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572393.770480:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.770487:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352d80 x1875444569523328/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.770509:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523328:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.770606:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.774858:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523328:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.774923:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.774935:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.774968:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.774986:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x9:0x0](ffff920a50d42a08) 00000080:00200000:1.0:1788572393.775070:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.775080:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-18, flags=0 00000080:00200000:1.0:1788572393.775085:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.775089:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-18, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.775126:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb44d lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be6e2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.775143:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x14:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.775192:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb44d lrc: 1/0,0 mode: --/PR res: [0x240002729:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be6e2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.775203:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.775206:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb44d lrc: 0/0,0 mode: --/PR res: [0x240002729:0x14:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be6e2 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572393.775227:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.775234:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350e00 x1875444569523584/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.775253:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523584:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.775315:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.779995:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523584:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.780069:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x14:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.780084:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.780116:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x14:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.780126:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x14:0x0](ffff920a50d41988) 00000080:00200000:1.0:1788572393.780215:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.780226:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-3, flags=0 00000080:00200000:1.0:1788572393.780231:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.780236:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-3, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.780273:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.780291:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x5:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.780346:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.780353:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.780357:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb3e4 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4d5 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572393.780380:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.780388:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351180 x1875444569523840/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.780410:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523840:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.780483:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.786704:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569523840:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.786780:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.786796:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.786833:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.786844:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x5:0x0](ffff920a50d4b248) 00000080:00200000:1.0:1788572393.786942:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.786958:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-16, flags=0 00000080:00200000:1.0:1788572393.786966:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.786982:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-16, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:1.0:1788572393.787048:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb43f lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be69c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.787068:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x12:0x0]: losing layout lock 00010000:00010000:1.0:1788572393.787116:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb43f lrc: 1/0,0 mode: --/PR res: [0x240002729:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be69c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.787124:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.787127:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb43f lrc: 0/0,0 mode: --/PR res: [0x240002729:0x12:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be69c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1788572393.787169:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.787178:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353b80 x1875444569524096/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.787203:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524096:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.788104:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8db40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.793880:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524096:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.793949:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x12:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.793963:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.794034:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x12:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.794044:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x12:0x0](ffff920a50d41148) 00000080:00200000:2.0:1788572393.794130:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.794142:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-4, flags=0 00000080:00200000:2.0:1788572393.794148:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.794153:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-4, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.794206:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.794225:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x6:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.794265:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 1/0,0 mode: --/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.794272:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.794276:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb3eb lrc: 0/0,0 mode: --/PR res: [0x240002729:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be4f8 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.794296:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.794304:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891500 x1875444569524352/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.794491:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524352:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.794562:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.798927:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524352:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.799008:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.799031:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.799059:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.799068:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x6:0x0](ffff920a50d3e3c8) 00000080:00200000:2.0:1788572393.799178:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.799190:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-15, flags=0 00000080:00200000:2.0:1788572393.799196:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.799200:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-15, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.799242:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb438 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be679 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.799263:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x11:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.799324:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb438 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be679 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.799332:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.799335:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb438 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x11:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be679 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.799360:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.799370:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52892680 x1875444569524608/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.799398:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524608:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.799469:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.808496:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524608:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.808836:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x11:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.808852:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.808881:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x11:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.808890:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x11:0x0](ffff920a50d40908) 00000080:00200000:2.0:1788572393.808985:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.808997:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-0, flags=0 00000080:00200000:2.0:1788572393.809002:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.809006:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-0, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.809285:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.809307:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x2:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.809355:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.809363:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.809365:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb3cf lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be46c expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.809388:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.809397:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891c00 x1875444569524864/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.809420:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524864:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.809503:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.817107:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569524864:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.817180:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.817195:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.817224:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.817232:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2:0x0](ffff920a50d4ec08) 00000080:00200000:2.0:1788572393.817335:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.817350:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-8, flags=0 00000080:00200000:2.0:1788572393.817357:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.817363:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-8, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.817405:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb407 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be584 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.817427:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xa:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.817476:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb407 lrc: 1/0,0 mode: --/PR res: [0x240002729:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be584 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.817483:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.817486:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb407 lrc: 0/0,0 mode: --/PR res: [0x240002729:0xa:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be584 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.817512:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.817520:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52892a00 x1875444569525120/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.817555:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525120:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.818010:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.822938:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525120:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.823027:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0xa:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.823041:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.823071:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0xa:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.823079:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0xa:0x0](ffff920a50d43a88) 00000080:00200000:2.0:1788572393.823178:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.823189:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-9, flags=0 00000080:00200000:2.0:1788572393.823195:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.823199:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-9, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.823238:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.823257:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xb:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.823303:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 1/0,0 mode: --/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.823311:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.823314:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78c00/0x8364c713e25bb40e lrc: 0/0,0 mode: --/PR res: [0x240002729:0xb:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5a7 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.823338:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.823347:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891f80 x1875444569525376/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.823369:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525376:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.823443:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.830501:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525376:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.830583:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0xb:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.830630:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.830655:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0xb:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.830663:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0xb:0x0](ffff920a50d4f448) 00000080:00200000:2.0:1788572393.830745:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.830757:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-11, flags=0 00000080:00200000:2.0:1788572393.830762:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.830767:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-11, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.830807:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.830827:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xd:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.830873:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 1/0,0 mode: --/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.830881:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.830884:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb41c lrc: 0/0,0 mode: --/PR res: [0x240002729:0xd:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5ed expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.830907:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.830916:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52893480 x1875444569525632/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.830940:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525632:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.831020:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.834515:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525632:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.834575:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0xd:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.834613:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.834637:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0xd:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.834645:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0xd:0x0](ffff920a50d4e3c8) 00000080:00200000:2.0:1788572393.834770:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.834779:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-19, flags=0 00000080:00200000:2.0:1788572393.834783:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.834787:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-19, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.834818:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.834836:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x15:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.834879:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.834887:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.834890:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb454 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x15:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be705 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.834910:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.834917:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890700 x1875444569525888/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.834937:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525888:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.835017:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036b40 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.838306:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569525888:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.838386:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x15:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.838399:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.838428:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x15:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.838436:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x15:0x0](ffff920a50d2ec08) 00000080:00200000:2.0:1788572393.838521:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.838834:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-17, flags=0 00000080:00200000:2.0:1788572393.838841:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.838847:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-17, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.838893:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb446 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be6bf expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.838911:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x13:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.838957:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb446 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be6bf expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.838963:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.838967:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb446 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x13:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be6bf expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.839001:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.839010:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52890e00 x1875444569526144/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.839033:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526144:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.839109:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.842894:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526144:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.842960:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.842983:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.843015:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.843023:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x13:0x0](ffff920a50d421c8) 00000080:00200000:2.0:1788572393.843106:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.843118:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=0 00000080:00200000:2.0:1788572393.843124:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.843129:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-1, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.843183:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000080:00200000:2.0:1788572393.843206:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.843249:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00010000:00010000:2.0:1788572393.843257:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.843260:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb3d6 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be48f expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 1 00000020:00100000:2.0:1788572393.843284:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.843293:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891880 x1875444569526400/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.843324:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526400:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.843393:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.847386:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526400:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.847449:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.847462:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.847490:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.847498:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3:0x0](ffff920a50d4d348) 00000080:00200000:2.0:1788572393.847635:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.847648:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-10, flags=0 00000080:00200000:2.0:1788572393.847654:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.847659:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-10, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.847710:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.847733:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xc:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.847792:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 1/0,0 mode: --/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.847799:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.847803:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb415 lrc: 0/0,0 mode: --/PR res: [0x240002729:0xc:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be5ca expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.847826:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.847834:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a52891180 x1875444569526656/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.847858:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526656:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.847930:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.851849:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526656:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.851912:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0xc:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.851927:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.851957:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0xc:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.851966:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0xc:0x0](ffff920a50d480c8) 00000080:00200000:2.0:1788572393.852061:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.852074:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-14, flags=0 00000080:00200000:2.0:1788572393.852080:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.852085:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-14, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.852125:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.852145:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x10:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.852204:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.852217:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.852220:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78e00/0x8364c713e25bb431 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x10:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be656 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.852244:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.852252:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6680 x1875444569526912/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.852275:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526912:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.852361:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036cc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.855457:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569526912:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.855515:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x10:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.855527:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.855565:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x10:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.855573:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x10:0x0](ffff920a50d29148) 00000080:00200000:2.0:1788572393.855737:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.855748:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-5, flags=0 00000080:00200000:2.0:1788572393.855753:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.855758:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-5, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.855796:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb3f2 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be51b expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.855815:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x7:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.855856:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb3f2 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be51b expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.855864:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.855868:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb3f2 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x7:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be51b expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.855892:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.855901:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5c00 x1875444569527168/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.855923:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527168:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572393.855999:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036f00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572393.860241:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527168:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572393.860307:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x7:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572393.860460:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572393.860485:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x7:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572393.860493:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x7:0x0](ffff920a50d47448) 00000080:00200000:2.0:1788572393.860620:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 1 00000080:00200000:2.0:1788572393.860634:0:162685:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-12, flags=0 00000080:00200000:2.0:1788572393.860639:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88), inode mode 41ed mask 3 00000080:00200000:2.0:1788572393.860645:0:162685:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-12, dir=[0x240002729:0x1:0x0](ffff920a50d4db88) 00010000:00010000:2.0:1788572393.860683:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb423 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109be610 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572393.860700:0:162685:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0xe:0x0]: losing layout lock 00010000:00010000:2.0:1788572393.860742:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb423 lrc: 1/0,0 mode: --/PR res: [0x240002729:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be610 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572393.860750:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572393.860753:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb423 lrc: 0/0,0 mode: --/PR res: [0x240002729:0xe:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109be610 expref: -99 pid: 162678 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572393.860783:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572393.860792:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6d80 x1875444569527424/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572393.860815:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6d80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527424:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.876625:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bd880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527680:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.876689:0:162685:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1788572393.876702:0:162685:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1788572393.876729:0:162685:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572393.876737:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x8:0x0](ffff920a50d45348) 00000080:00200008:1.0:1788572393.876824:0:162685:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x1:0x0](ffff920a50d4db88), flags 704000 00010000:00010000:1.0:1788572393.876851:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572393.876890:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.876907:0:162685:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), flags=0 00000080:00200000:1.0:1788572393.876912:0:162685:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08), intent=getattr 00000002:00010000:1.0:1788572393.876953:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x2:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572393.877011:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.877020:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.877030:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.877054:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527808:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.877126:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d180 going to sleep for 11 seconds 00010000:00010000:2.0:1788572393.885754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572393.885925:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527808:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.885952:0:162685:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.885971:0:162685:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.885993:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886004:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886015:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886027:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886035:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886042:0:162685:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.886061:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48351880 x1875444569527808/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.886074:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886087:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.886098:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.886106:0:162685:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886113:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886120:0:162685:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886149:0:162685:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:1.0:1788572393.886174:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240002729:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572393.886222:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886230:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.886244:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.886266:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527936:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.886359:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00010000:00010000:3.0:1788572393.886568:0:160929:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:3.0:1788572393.886575:0:160929:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886608:0:160929:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78000 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572393.886615:0:160929:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886623:0:160929:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886632:0:160929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886649:0:160929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572393.886680:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 2/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886687:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572393.886705:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886713:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb48c lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be9cf expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572393.886726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572393.886728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572393.886730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572393.886743:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569528064:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572393.891732:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569528064:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572393.891839:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569527936:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.891861:0:162685:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891873:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891885:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891896:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891910:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891918:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891925:0:162685:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.891942:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48352300 x1875444569527936/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.891955:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.891969:0:162685:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/1 (0 0) 00000080:00200000:1.0:1788572393.891995:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x2:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.892026:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x1:0x0], updating i_size 4096 00000080:00200000:1.0:1788572393.892030:0:162685:0:(namei.c:147:ll_iget()) got inode: [0x240002729:0x1:0x0](ffff920a50d4db88): rc = 0 00000080:00010000:1.0:1788572393.892036:0:162685:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for remote lock 0x8364c713e25bb493 00000080:00010000:1.0:1788572393.892041:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb3c1 00000080:00010000:1.0:1788572393.892051:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00010000:1.0:1788572393.892054:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3c1 from it ffffaf73018dbe00 00010000:00010000:1.0:1788572393.892058:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.892068:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:1.0:1788572393.892082:0:162685:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x8364c713e25bb493 from it ffffaf73018dbe00 00010000:00010000:1.0:1788572393.892085:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.892092:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.892115:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.892123:0:162685:0:(namei.c:2261:ll_rmdir()) VFS Op:name=remote_dir, dir=[0x200027501:0x2:0x0](ffff920a50d3cb08) 00010000:00010000:1.0:1788572393.892149:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.892259:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.892267:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572393.892876:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb3c1 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be434 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.892905:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.892922:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1788572393.892956:0:162685:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48350000 x1875444569522560/t21474837526(21474837526) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.892967:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.892983:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351880 x1875444569528448/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.892998:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528448:192.168.202.157@tcp:35:rm.0 00000100:00100000:1.0:1788572393.893026:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.895891:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528448:192.168.202.157@tcp:35:rm.0 00000002:00100000:1.0:1788572393.895932:0:162685:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48350000 x1875444569522560/t21474837526(21474837526) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572393.896026:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.896035:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 1/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572393.896040:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788572393.896044:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb493 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be9dd expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.896054:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb485 lrc: 0/0,0 mode: --/CR res: [0x240002729:0x1:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be77c expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000020:00100000:1.0:1788572393.896074:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.896080:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351880 x1875444569528704/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.896101:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528704:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.896170:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.898346:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569528192:192.168.202.157@tcp:103:rm.0 00000100:00100000:3.0:1788572393.900747:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572393.900847:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584387072 00000100:00100000:2.0:1788572393.900857:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584387072 00000100:00100000:2.0:1788572393.900914:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 4 00000100:00100000:2.0:1788572393.900924:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584387072:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572393.900943:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78400/0x8364c713e25bb3ba lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109be418 expref: -99 pid: 162673 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572393.901029:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572393.901037:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584387072:12345-192.168.202.157@tcp:104: Request processed in 114us (298us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572393.901044:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 4 00000100:00100000:1.0:1788572393.914729:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528704:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.914776:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x1:0x0](ffff920a50d4db88) 00000080:00200008:1.0:1788572393.914840:0:162685:0:(file.c:479:ll_file_release()) START file d80a.replay-single:[0x200027501:0x2:0x0](ffff920a50d3cb08), flags 704000 00010000:00010000:1.0:1788572393.914880:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572393.914922:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.914939:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.914950:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788572393.914959:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:1.0:1788572393.914967:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.914982:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.914986:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.914993:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf73018dbc10 00010000:00010000:1.0:1788572393.914997:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:1.0:1788572393.915005:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1788572393.915020:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572393.915038:0:162685:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=0 00000080:00200000:1.0:1788572393.915042:0:162685:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:1.0:1788572393.915055:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80a.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572393.915095:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.915102:0:162685:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.915112:0:162685:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 162685 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572393.915130:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528960:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572393.915209:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.919426:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569528960:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572393.919451:0:162685:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919466:0:162685:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919474:0:162685:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919485:0:162685:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919496:0:162685:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919508:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919516:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919528:0:162685:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572393.919552:0:162685:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48352300 x1875444569528960/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572393.919567:0:162685:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919577:0:162685:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/2 (0 0) 00000080:00200000:1.0:1788572393.919583:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572393.919657:0:162685:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x2:0x0], updating i_size 4096 00000080:00200000:1.0:1788572393.919660:0:162685:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x2:0x0](ffff920a50d3cb08): rc = 0 00000080:00010000:1.0:1788572393.919668:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) for lock 0x8364c713e25bb49a 00000080:00010000:1.0:1788572393.919678:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x2:0x0](ffff920a50d3cb08) 00000080:00010000:1.0:1788572393.919682:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb49a from it ffffaf73018dbe00 00010000:00010000:1.0:1788572393.919685:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572393.919693:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572393.919726:0:162685:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572393.919743:0:162685:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572393.919753:0:162685:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1788572393.919769:0:162685:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:1.0:1788572393.919776:0:162685:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572393.919780:0:162685:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572393.919783:0:162685:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb3b3 00000080:00010000:1.0:1788572393.919786:0:162685:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb3b3 from it ffffaf73018dbd88 00010000:00010000:1.0:1788572393.919789:0:162685:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:1.0:1788572393.919796:0:162685:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:1.0:1788572393.919804:0:162685:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572393.919810:0:162685:0:(namei.c:2261:ll_rmdir()) VFS Op:name=d80a.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:1.0:1788572393.919831:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1788572393.919853:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.919864:0:162685:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:1.0:1788572393.919903:0:162685:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a52893b80 x1875444569521536/t347892350983(347892350983) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572393.919919:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572393.919925:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352300 x1875444569529216/t0(0) o35->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572393.919945:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569529216:192.168.202.157@tcp:35:rm.0 00000100:00100000:1.0:1788572393.920027:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.922798:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569529216:192.168.202.157@tcp:35:rm.0 00000002:00100000:1.0:1788572393.922833:0:162685:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a52893b80 x1875444569521536/t347892350983(347892350983) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572409 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572393.922906:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 1/0,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.922917:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 1/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572393.922925:0:162685:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:1.0:1788572393.922931:0:162685:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:1.0:1788572393.922934:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb49a lrc: 0/0,0 mode: --/PR res: [0x200027501:0x2:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bea2a expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572393.922947:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb470 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x2:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be74b expref: -99 pid: 162685 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572393.922961:0:162685:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb3b3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109be3ee expref: -99 pid: 162187 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000020:00100000:1.0:1788572393.922996:0:162685:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572393.923002:0:162685:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352300 x1875444569529472/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572393.923021:0:162685:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569529472:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572393.923088:0:162685:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572393.927253:0:162685:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:162685:1875444569529472:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572393.927300:0:162685:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x2:0x0](ffff920a50d3cb08) 00010000:00010000:2.0:1788572394.915374:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572396.961576:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572397.532029:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569529600:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572397.532142:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529856:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572397.532168:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569530112:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572397.532198:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572397.532204:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529728:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572397.532226:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529984:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572397.541888:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529728:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572397.541923:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529984:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572397.541932:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569529600:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572397.541942:0:2400:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 21474837548 gen 1 00000100:00100000:2.0:1788572397.542157:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569529856:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572397.542172:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569530112:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572397.983962:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572399.003729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572400.028687:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572400.699258:0:163253:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 21:39:59 (1788572399) 00010000:00010000:3.0:1788572401.051686:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1788572402.050952:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572402.050981:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572402.051042:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.051053:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.051063:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572402.051088:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6d80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569530752:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572402.053713:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572402.064061:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6d80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569530752:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572402.064086:0:163277:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064098:0:163277:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064110:0:163277:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064120:0:163277:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064131:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064151:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064159:0:163277:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572402.064170:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac6d80 x1875444569530752/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572402.064183:0:163277:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064197:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572402.064205:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572402.064233:0:163277:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1788572402.064239:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:2.0:1788572402.064245:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572402.064248:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301d67b58 00010000:00010000:2.0:1788572402.064251:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064260:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572402.064290:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572402.064308:0:163277:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=29 00000080:00200000:2.0:1788572402.064312:0:163277:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80b.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:2.0:1788572402.064849:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80b.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572402.064896:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064905:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572402.064913:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572402.064933:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6d80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569530880:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572402.067128:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.072828:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6d80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569530880:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572402.072857:0:163277:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072879:0:163277:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x4:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072887:0:163277:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072897:0:163277:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072909:0:163277:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072922:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072929:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072935:0:163277:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572402.072945:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a60ac6d80 x1875444569530880/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572402.072959:0:163277:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.072968:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/4 (0 0) 00000080:00200000:3.0:1788572402.072973:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572402.073001:0:163277:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x4:0x0], updating i_size 4096 00000080:00200000:3.0:1788572402.073004:0:163277:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x4:0x0](ffff920a50d442c8): rc = 0 00000080:00010000:3.0:1788572402.073009:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x4:0x0](ffff920a50d442c8) for lock 0x8364c713e25bb4b6 00000080:00010000:3.0:1788572402.073019:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x4:0x0](ffff920a50d442c8) 00000080:00010000:3.0:1788572402.073022:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4b6 from it ffffaf7301d67a28 00010000:00010000:3.0:1788572402.073026:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.073034:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb4b6 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beaee expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572402.073070:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572402.073083:0:163277:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x4:0x0](ffff920a50d442c8), flags=13 00000080:00200000:3.0:1788572402.073087:0:163277:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x4:0x0](ffff920a50d442c8), intent=getattr 00000002:00010000:3.0:1788572402.073109:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x4:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572402.073161:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.073169:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.073179:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572402.073200:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531008:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572402.073278:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645300 going to sleep for 11 seconds 00010000:00010000:3.0:1788572402.077877:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572402.078104:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531008:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572402.078127:0:163277:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078152:0:163277:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078160:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078167:0:163277:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 3/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078176:0:163277:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078182:0:163277:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec7e00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572402.078186:0:163277:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078198:0:163277:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078204:0:163277:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 5/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078211:0:163277:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572402.078217:0:163277:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 3/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078223:0:163277:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7e00/0x8364c713e25bb4bd lrc: 0/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572402.078238:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a51b3c700 x1875444569531008/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572402.078251:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x4:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572402.078275:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/4 (0 0) 00000080:00200000:1.0:1788572402.078353:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.078370:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.078380:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572402.078393:0:163277:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572402.078400:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.078406:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.078410:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.078417:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301d67be8 00010000:00010000:1.0:1788572402.078420:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572402.078430:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572402.078443:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572402.078451:0:163277:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=16408 00000080:00200000:1.0:1788572402.078456:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572402.078466:0:163277:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x4:0x0](ffff920a50d442c8), flags=16392 00000080:00200000:1.0:1788572402.078471:0:163277:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x4:0x0](ffff920a50d442c8), intent=getattr 00000002:00010000:1.0:1788572402.078481:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x4:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572402.078521:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078528:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.078537:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572402.078556:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531136:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572402.078655:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572402.088558:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531136:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572402.088614:0:163277:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088636:0:163277:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088644:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088650:0:163277:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088659:0:163277:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088664:0:163277:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572402.088668:0:163277:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088676:0:163277:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088682:0:163277:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 5/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088689:0:163277:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572402.088696:0:163277:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.088703:0:163277:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb4c4 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572402.088720:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a48350000 x1875444569531136/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572402.088735:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x4:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572402.088881:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/4 (0 0) 00000080:00200000:1.0:1788572402.088948:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.088963:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.088972:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572402.088984:0:163277:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572402.088991:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.088996:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.088999:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.089004:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301d67b58 00010000:00010000:1.0:1788572402.089008:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572402.089017:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572402.089030:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572402.089038:0:163277:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=13 00000080:00200008:1.0:1788572402.089047:0:163277:0:(file.c:6296:ll_getattr_dentry()) START file d80b.replay-single:[0x200027501:0x4:0x0](ffff920a50d442c8), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572402.089052:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8),name=d80b.replay-single 00000002:00010000:1.0:1788572402.089064:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.089070:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/4 (0 0) 00010000:00010000:1.0:1788572402.089104:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.089111:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.089121:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572402.089152:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531264:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572402.089221:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572402.092154:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531264:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572402.092179:0:163277:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092196:0:163277:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: --/PR res: [0x200027501:0x4:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092203:0:163277:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: --/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092214:0:163277:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092225:0:163277:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092237:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092245:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092257:0:163277:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572402.092269:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48350000 x1875444569531264/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572402.092282:0:163277:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092293:0:163277:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/4 (0 0) 00000080:00200000:1.0:1788572402.092299:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572402.092441:0:163277:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x4:0x0], updating i_size 4096 00000080:00010000:1.0:1788572402.092449:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x4:0x0](ffff920a50d442c8) for lock 0x8364c713e25bb4cb 00000080:00010000:1.0:1788572402.092456:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x4:0x0](ffff920a50d442c8) 00000080:00010000:1.0:1788572402.092459:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4cb from it ffffaf7301d67cb8 00010000:00010000:1.0:1788572402.092463:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.092473:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572402.094179:0:163277:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572402.094188:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.094207:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.094225:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572402.094246:0:163277:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572402.094255:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.094263:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.094267:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.094275:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301d67cb8 00010000:00010000:1.0:1788572402.094279:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572402.094288:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572402.098908:0:163277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.098937:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.098956:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572402.098973:0:163277:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572402.098981:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.098989:0:163277:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.098992:0:163277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.099000:0:163277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301d67bf8 00010000:00010000:1.0:1788572402.099005:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572402.099015:0:163277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572402.099032:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572402.099040:0:163277:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=267 00000080:00200000:1.0:1788572402.099046:0:163277:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 44 00000080:00200008:1.0:1788572402.099053:0:163277:0:(file.c:1090:ll_file_open()) START file d80b.replay-single:[0x200027501:0x4:0x0](ffff920a50d442c8), flags 300000 00000002:00010000:1.0:1788572402.099068:0:163277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788572402.099116:0:163277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.099124:0:163277:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572402.099153:0:163277:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572402.099165:0:163277:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350000 x1875444569531520/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572402.099176:0:163277:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572402.099197:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531520:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572402.099609:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572402.106058:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531520:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572402.106079:0:163277:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106092:0:163277:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106100:0:163277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106107:0:163277:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106117:0:163277:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106122:0:163277:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79000 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572402.106126:0:163277:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106133:0:163277:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106152:0:163277:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106159:0:163277:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572402.106166:0:163277:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572402.106173:0:163277:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb4d2 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 163277 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572402.106190:0:163277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a48350000 x1875444569531520/t347892350988(347892350988) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572418 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572402.106204:0:163277:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569531520/t347892350988(347892350988) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572418 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572402.106221:0:163277:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572402.106248:0:163277:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x4:0x0], updating i_size 4096 00000002:00100000:1.0:1788572402.106265:0:163277:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569531520/t347892350988(347892350988) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572418 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572402.106347:0:163277:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8) cmd=c03066f0 arg=7ffdc4c0fd40 00000080:00200000:1.0:1788572402.106363:0:163277:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00000020:00100000:1.0:1788572402.106422:0:163277:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572402.106430:0:163277:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351c00 x1875444569531776/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572402.106451:0:163277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569531776:192.168.202.157@tcp:36:lfs.0 00000100:00100000:1.0:1788572402.106522:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.111452:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572402.111490:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584394752 00000100:00100000:3.0:1788572402.111500:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584394752 00000100:00100000:3.0:1788572402.111548:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 5 00000100:00100000:3.0:1788572402.111557:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188054:x1875444584394752:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572402.111575:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb4cb lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109beb26 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572402.111685:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572402.111692:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188054:x1875444584394752:12345-192.168.202.157@tcp:104: Request processed in 137us (249us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572402.111709:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 5 00000100:00100000:1.0:1788572402.112447:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569531904:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572402.120252:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478be300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569531904:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1788572402.652920:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532160:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572402.653019:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572402.653053:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572402.653059:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569532032:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572402.653090:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532288:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572402.653114:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532544:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572402.657124:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569532032:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572402.657157:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532288:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572402.657171:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532544:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572402.657180:0:2400:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350988 gen 1 00000100:00100000:3.0:1788572402.657200:0:2400:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff920a48350000 x1875444569531520/t347892350988(347892350988) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572418 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572402.657209:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532160:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572402.657221:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569532416:192.168.202.157@tcp:400:kworker.0 00000080:00200008:3.0:1788572402.783916:0:163286:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572402.783932:0:163286:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.783958:0:163286:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.783978:0:163286:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572402.783994:0:163286:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572402.784003:0:163286:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.784011:0:163286:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.784015:0:163286:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:3.0:1788572402.784022:0:163286:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301c37cb8 00010000:00010000:3.0:1788572402.784027:0:163286:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572402.784037:0:163286:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:3.0:1788572402.788400:0:163286:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572402.788413:0:163286:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.788437:0:163286:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.788456:0:163286:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572402.788472:0:163286:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572402.788481:0:163286:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.788489:0:163286:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.788492:0:163286:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:3.0:1788572402.788500:0:163286:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301c37cb8 00010000:00010000:3.0:1788572402.788503:0:163286:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572402.788513:0:163286:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572402.790875:0:163286:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.790906:0:163286:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.790926:0:163286:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572402.790945:0:163286:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00000080:00200000:3.0:1788572402.790954:0:163286:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.790962:0:163286:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.790966:0:163286:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:3.0:1788572402.790973:0:163286:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7301c37bb0 00010000:00010000:3.0:1788572402.790977:0:163286:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 1 00010000:00010000:3.0:1788572402.790986:0:163286:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572402.791003:0:163286:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572402.791014:0:163286:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:3.0:1788572402.791138:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.791172:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00800000:00000004:3.0:1788572402.791180:0:163286:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9151, max_age 9160 00000100:00100000:3.0:1788572402.791227:0:163286:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532672:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572402.791324:0:163286:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.802393:0:163286:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532672:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572402.802422:0:163286:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023635/1024000 00000080:00200080:3.0:1788572402.802508:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.802525:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00800000:00000004:3.0:1788572402.802533:0:163286:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9151, max_age 9160 00000100:00100000:3.0:1788572402.802572:0:163286:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532800:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572402.802681:0:163286:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.811354:0:163286:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 21474837549 gen 1 00000100:00100000:3.0:1788572402.811374:0:163286:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532800:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572402.811402:0:163286:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00200080:3.0:1788572402.811475:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811493:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811501:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811506:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811511:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811514:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811518:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811522:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811527:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811531:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811535:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811539:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811543:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811548:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811552:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811556:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811560:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811564:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811568:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811573:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811577:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811581:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811627:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811633:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811638:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811642:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811646:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811650:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811654:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811658:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811662:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811666:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811670:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811673:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811677:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811681:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811685:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811688:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811692:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811696:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811699:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811703:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811706:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811709:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811713:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811728:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811734:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811737:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811741:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811745:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811749:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811752:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811756:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811760:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811764:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811768:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811771:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811775:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811779:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811782:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811786:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811790:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811794:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811797:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811801:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811804:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811807:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811811:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811815:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811819:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811823:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811826:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811830:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811834:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811837:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811842:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811845:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811849:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811853:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811856:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811860:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811863:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811867:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811870:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811874:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811878:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811882:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811886:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811891:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811894:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811899:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811902:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811906:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811910:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811915:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811920:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811924:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811928:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811931:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811934:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811938:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811942:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811946:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811949:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811953:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811956:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811960:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811963:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811967:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811970:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811974:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811977:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811981:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811985:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811989:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.811993:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.811997:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.812000:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.812004:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.812008:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.812012:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.812015:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.812019:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.812022:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.812026:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00800000:00000080:3.0:1788572402.812030:0:163286:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.812034:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.812040:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00020000:00000004:3.0:1788572402.812048:0:163286:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9151, max_age 9160 00000100:00100000:3.0:1788572402.812084:0:163286:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532928:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572402.812173:0:163286:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.820346:0:163286:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569532928:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572402.820382:0:163286:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261712/262144 00000080:00200080:3.0:1788572402.820473:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.820491:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00020000:00000004:3.0:1788572402.820498:0:163286:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9151, max_age 9160 00000100:00100000:3.0:1788572402.820537:0:163286:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569533056:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572402.820636:0:163286:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.826858:0:163286:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163286:1875444569533056:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572402.826884:0:163286:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261737/262144 00000080:00200080:3.0:1788572402.826951:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.826972:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.826980:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.826986:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.826990:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.826994:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.826998:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827001:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827005:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827008:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827013:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827016:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827020:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827023:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827028:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827031:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827035:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827039:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827043:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827045:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827047:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827049:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827051:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827053:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827055:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827056:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827058:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827060:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827063:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827065:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827067:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827069:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827071:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827073:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827075:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827076:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827078:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827080:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827082:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827083:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827085:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827087:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827089:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827090:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827092:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827094:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827096:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827098:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827099:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827101:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827103:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827105:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827107:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827109:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827111:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827114:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827116:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827118:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827120:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827121:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827123:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827125:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827127:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827128:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827130:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827132:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827134:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827136:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827137:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827139:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827141:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827143:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827145:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827146:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827148:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827162:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827167:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827171:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827176:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827183:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827188:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827192:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827196:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827200:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827204:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827207:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827211:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827215:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827219:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827222:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827226:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827230:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827234:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827238:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827242:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827246:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827250:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827253:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827258:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827260:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827264:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827267:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827271:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827275:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827279:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827282:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827286:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827289:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827292:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827295:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827299:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827302:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827305:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827308:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827311:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827315:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827318:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827322:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827326:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827330:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827334:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827669:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827677:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827680:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200080:3.0:1788572402.827684:0:163286:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff0e1eb0f0 00020000:00000080:3.0:1788572402.827687:0:163286:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a54090800 uarg=0000000000000000 00000080:00200008:3.0:1788572402.827701:0:163286:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:1.0:1788572402.900675:0:163293:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.900730:0:163293:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.900753:0:163293:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572402.900771:0:163293:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:1.0:1788572402.900780:0:163293:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.900789:0:163293:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.900793:0:163293:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.900802:0:163293:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7300c8fbd0 00010000:00010000:1.0:1788572402.900806:0:163293:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:1.0:1788572402.900815:0:163293:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572402.900832:0:163293:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572402.900853:0:163293:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=1536 00000080:00200000:1.0:1788572402.900858:0:163293:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.900865:0:163293:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.900872:0:163293:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572402.900882:0:163293:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:1.0:1788572402.900888:0:163293:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.900892:0:163293:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.900895:0:163293:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.900898:0:163293:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7300c8fcd0 00010000:00010000:1.0:1788572402.900901:0:163293:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:1.0:1788572402.900908:0:163293:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572402.900917:0:163293:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572402.900925:0:163293:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572402.900931:0:163293:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572402.900937:0:163293:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572402.900945:0:163293:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:1.0:1788572402.900951:0:163293:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572402.900955:0:163293:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572402.900957:0:163293:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4af 00000080:00010000:1.0:1788572402.900961:0:163293:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4af from it ffffaf7300c8fdb0 00010000:00010000:1.0:1788572402.900964:0:163293:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1788572402.900971:0:163293:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:1.0:1788572402.900980:0:163293:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572402.901441:0:163293:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=33188, excl=1 00000080:00200000:1.0:1788572402.901448:0:163293:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) mode 100644 dev 0 00010000:00010000:1.0:1788572402.901484:0:163293:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572402.901537:0:163293:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1788572402.901544:0:163293:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572402.901548:0:163293:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb4af lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bead2 expref: -99 pid: 163277 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:1.0:1788572402.901576:0:163293:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572402.901616:0:163293:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352a00 x1875444569533312/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572402.901646:0:163293:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163293:1875444569533312:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:1.0:1788572402.902498:0:163293:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528960c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572402.910119:0:163293:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163293:1875444569533312:192.168.202.157@tcp:36:mcreate.0 00000080:00200000:1.0:1788572402.910182:0:163293:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x5:0x0](ffff920a50d4db88) 00000080:00200000:1.0:1788572402.910228:0:163293:0:(namei.c:2070:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, unhashed 1 00000080:00200000:3.0:1788572402.946231:0:163296:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.946262:0:163296:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.946324:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.946335:0:163296:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.946346:0:163296:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572402.946373:0:163296:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533440:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572402.946460:0:163296:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.969285:0:163296:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533440:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572402.969311:0:163296:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969324:0:163296:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969337:0:163296:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969566:0:163296:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969581:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969628:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969636:0:163296:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572402.969648:0:163296:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3dc00 x1875444569533440/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572402.969662:0:163296:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969674:0:163296:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572402.969681:0:163296:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572402.969708:0:163296:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572402.969770:0:163296:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4d9 00000080:00010000:3.0:1788572402.969778:0:163296:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.969781:0:163296:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4d9 from it ffffaf7300c97b58 00010000:00010000:3.0:1788572402.969784:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969794:0:163296:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572402.969821:0:163296:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572402.969842:0:163296:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:3.0:1788572402.969846:0:163296:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572402.969867:0:163296:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572402.969907:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969915:0:163296:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.969924:0:163296:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163296 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572402.969944:0:163296:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533568:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572402.970019:0:163296:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.980198:0:163296:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533568:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572402.980226:0:163296:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980248:0:163296:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x5:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980256:0:163296:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980266:0:163296:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980469:0:163296:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980484:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980497:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980505:0:163296:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572402.980517:0:163296:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3dc00 x1875444569533568/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572402.980531:0:163296:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980542:0:163296:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/5 (0 0) 00000080:00200000:3.0:1788572402.980549:0:163296:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572402.980574:0:163296:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x5:0x0], updating i_size 0 00000080:00200000:3.0:1788572402.980576:0:163296:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x5:0x0](ffff920a50d4db88): rc = 0 00000080:00200000:3.0:1788572402.980677:0:163296:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x5:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572402.980683:0:163296:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x5:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb4e0 00000080:00010000:3.0:1788572402.980694:0:163296:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x5:0x0](ffff920a50d4db88) 00000080:00010000:3.0:1788572402.980698:0:163296:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e0 from it ffffaf7300c97b20 00010000:00010000:3.0:1788572402.980701:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572402.980710:0:163296:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572402.980751:0:163296:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x200027501:0x5:0x0](ffff920a50d4db88), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572402.980759:0:163296:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x5:0x0](ffff920a50d4db88),name=fsa-oleg257-client.virtnet 00000002:00010000:3.0:1788572402.980781:0:163296:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x5:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.980790:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572402.980802:0:163296:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572402.980809:0:163296:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.980814:0:163296:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x5:0x0](ffff920a50d4db88) 00000080:00010000:3.0:1788572402.980817:0:163296:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x5:0x0](ffff920a50d4db88) for lock 0x8364c713e25bb4e0 00000080:00010000:3.0:1788572402.980820:0:163296:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e0 from it ffffaf7300c97cb8 00010000:00010000:3.0:1788572402.980823:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572402.980830:0:163296:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572402.980889:0:163296:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.980898:0:163296:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.980905:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572402.980913:0:163296:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572402.980919:0:163296:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.980923:0:163296:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.980925:0:163296:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4d9 00000080:00010000:3.0:1788572402.980929:0:163296:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4d9 from it ffffaf7300c97c10 00010000:00010000:3.0:1788572402.980932:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572402.980939:0:163296:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572402.980947:0:163296:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572402.980957:0:163296:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:3.0:1788572402.980962:0:163296:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572402.980968:0:163296:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572402.980973:0:163296:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572402.980980:0:163296:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572402.980986:0:163296:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572402.980990:0:163296:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572402.980992:0:163296:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4d9 00000080:00010000:3.0:1788572402.980995:0:163296:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4d9 from it ffffaf7300c97d68 00010000:00010000:3.0:1788572402.980998:0:163296:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572402.981004:0:163296:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572402.981016:0:163296:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572402.981021:0:163296:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:3.0:1788572402.981041:0:163296:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572402.981065:0:163296:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572402.981074:0:163296:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x5:0x0]: losing layout lock 00010000:00010000:3.0:1788572402.981106:0:163296:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572402.981114:0:163296:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572402.981120:0:163296:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572402.981123:0:163296:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6a00/0x8364c713e25bb4e0 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x5:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bebc7 expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572402.981142:0:163296:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4d9 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bebab expref: -99 pid: 163296 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572402.981168:0:163296:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572402.981176:0:163296:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3dc00 x1875444569533824/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572402.981198:0:163296:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533824:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572402.982692:0:163296:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645c00 going to sleep for 11 seconds 00000100:00100000:3.0:1788572402.994364:0:163296:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163296:1875444569533824:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572402.994505:0:163296:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x5:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572402.994515:0:163296:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x5:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572402.994522:0:163296:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x5:0x0](ffff920a50d4db88) 00010000:00010000:3.0:1788572403.100085:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572404.127509:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572405.150577:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572406.176078:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572407.199056:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572407.778436:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534080:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.778524:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534336:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.778559:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788572407.778565:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569533952:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.778619:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534208:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572407.778642:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572407.805303:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534208:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572407.805550:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569533952:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572407.805564:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534336:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572407.821029:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572407.821049:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569534080:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572408.219762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572410.267868:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572410.497255:0:163377:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572410.497270:0:163377:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572410.497292:0:163377:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572410.497376:0:163377:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 163377 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.497387:0:163377:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163377 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.497399:0:163377:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163377 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572410.497427:0:163377:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534592:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572410.497574:0:163377:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528963c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572410.507209:0:163377:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534592:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572410.507241:0:163377:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.507254:0:163377:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.507268:0:163377:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.507281:0:163377:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.507295:0:163377:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.508048:0:163377:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.508189:0:163377:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572410.508206:0:163377:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48352680 x1875444569534592/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572426 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572410.508220:0:163377:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572410.508240:0:163377:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572410.508247:0:163377:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572410.509544:0:163377:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1788572410.509559:0:163377:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:2.0:1788572410.509567:0:163377:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572410.509570:0:163377:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301d7fcb8 00010000:00010000:2.0:1788572410.509575:0:163377:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572410.509614:0:163377:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572410.510719:0:163377:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572410.510727:0:163377:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572410.510821:0:163377:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572410.510840:0:163377:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572410.510855:0:163377:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572410.510863:0:163377:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572410.510870:0:163377:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572410.510874:0:163377:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:2.0:1788572410.510879:0:163377:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301d7fcb8 00010000:00010000:2.0:1788572410.510882:0:163377:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572410.510890:0:163377:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572410.511010:0:163377:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572410.511020:0:163377:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572410.511027:0:163377:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572410.511036:0:163377:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572410.511042:0:163377:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572410.511047:0:163377:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572410.511049:0:163377:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:2.0:1788572410.511053:0:163377:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301d7fbb0 00010000:00010000:2.0:1788572410.511056:0:163377:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572410.511064:0:163377:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572410.511073:0:163377:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572410.511080:0:163377:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:2.0:1788572410.511171:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.511186:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00800000:00000004:2.0:1788572410.511192:0:163377:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9161, max_age 9168 00000100:00100000:2.0:1788572410.511231:0:163377:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534720:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572410.511457:0:163377:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e49c0 going to sleep for 50 seconds 00000100:00100000:2.0:1788572410.517705:0:163377:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534720:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572410.517738:0:163377:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023635/1024000 00000080:00200080:2.0:1788572410.517822:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.517840:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00800000:00000004:2.0:1788572410.517848:0:163377:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9161, max_age 9168 00000100:00100000:2.0:1788572410.517886:0:163377:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534848:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572410.521742:0:163377:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572410.525522:0:163377:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534848:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572410.525557:0:163377:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317327/353529 objects 1023723/1024000 00000080:00200080:2.0:1788572410.525665:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525684:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525692:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525697:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525701:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525705:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525709:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525713:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525717:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525721:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525727:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525730:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525734:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525738:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525742:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525746:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525750:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525753:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525757:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525761:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525765:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525768:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525772:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525775:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525779:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525782:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525785:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525788:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525792:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525796:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525800:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525804:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525807:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525811:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525815:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525819:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525822:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525826:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525829:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525833:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525836:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525840:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525843:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525846:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525850:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525854:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525858:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525862:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525865:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525868:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525872:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525875:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525879:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525883:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525886:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525889:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525893:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525896:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525899:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525903:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525907:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525910:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525915:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525918:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525923:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525926:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525930:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525933:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525937:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525940:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525944:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525947:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525950:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525953:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525957:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525961:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525965:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525969:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525973:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525977:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525981:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525984:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525988:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525991:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.525995:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.525999:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526003:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526006:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526010:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526013:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526016:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526019:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526023:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526026:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526030:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526033:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526037:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526040:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526044:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526048:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526051:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526055:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526446:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526454:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526459:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526463:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526466:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526470:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526474:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526477:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526482:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526486:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526490:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526494:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526498:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526502:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526506:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526509:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526527:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526532:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526536:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526540:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526545:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526549:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526553:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00800000:00000080:2.0:1788572410.526557:0:163377:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.526561:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.526569:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00020000:00000004:2.0:1788572410.526578:0:163377:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9161, max_age 9168 00000100:00100000:2.0:1788572410.526710:0:163377:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534976:192.168.202.157@tcp:13:lfs.0 00000100:00100000:2.0:1788572410.526799:0:163377:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572410.530280:0:163377:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569534976:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572410.530464:0:163377:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261721/262144 00000080:00200080:2.0:1788572410.530546:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.530564:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00020000:00000004:2.0:1788572410.530571:0:163377:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9161, max_age 9168 00000100:00100000:2.0:1788572410.530645:0:163377:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569535104:192.168.202.157@tcp:13:lfs.0 00000100:00100000:2.0:1788572410.530715:0:163377:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e49c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572410.533561:0:163377:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163377:1875444569535104:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572410.533777:0:163377:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261737/262144 00000080:00200080:2.0:1788572410.533847:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534019:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534030:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534036:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534040:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534045:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534049:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534053:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534066:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534071:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534076:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534080:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534084:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534087:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534091:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534095:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534100:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534103:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534110:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534113:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534117:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534121:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534125:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534128:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534132:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534135:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534139:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534142:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534146:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534149:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534152:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534155:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534159:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534163:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534166:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534169:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534173:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534177:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534181:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534184:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534188:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534191:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534195:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534200:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534203:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534207:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534210:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534213:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534216:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534219:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534223:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534226:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534230:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534233:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534236:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534239:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534242:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534245:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534248:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534251:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534255:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534258:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534261:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534264:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534268:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534271:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534275:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534278:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534282:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534285:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534289:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534292:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534297:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534300:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534576:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534583:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534667:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534673:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534677:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534681:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534685:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534689:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534693:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534696:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534700:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534703:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534707:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534710:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534714:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534718:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534722:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534725:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534728:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534732:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534735:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534739:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534742:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534746:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534750:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534754:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534758:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534762:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534765:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534768:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534771:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534775:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534778:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534781:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534785:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534789:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534792:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534796:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534800:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534804:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534808:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534812:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534815:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534819:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534823:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534827:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534831:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534835:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534838:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534841:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200080:2.0:1788572410.534845:0:163377:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffe4692f720 00020000:00000080:2.0:1788572410.534848:0:163377:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45850c00 uarg=0000000000000000 00000080:00200008:2.0:1788572410.534932:0:163377:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:3.0:1788572410.613741:0:163384:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572410.613774:0:163384:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.613794:0:163384:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572410.613811:0:163384:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572410.613819:0:163384:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.613825:0:163384:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.613828:0:163384:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:3.0:1788572410.613835:0:163384:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301dc7bd0 00010000:00010000:3.0:1788572410.613838:0:163384:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572410.613845:0:163384:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572410.613857:0:163384:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572410.613867:0:163384:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=1536 00000080:00200000:3.0:1788572410.613870:0:163384:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572410.613876:0:163384:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.613882:0:163384:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572410.613890:0:163384:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572410.613897:0:163384:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.613901:0:163384:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.613904:0:163384:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:3.0:1788572410.613907:0:163384:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301dc7cd0 00010000:00010000:3.0:1788572410.613911:0:163384:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572410.613919:0:163384:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572410.613927:0:163384:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572410.613932:0:163384:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572410.613937:0:163384:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.613941:0:163384:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572410.613946:0:163384:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572410.613950:0:163384:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.613952:0:163384:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.613954:0:163384:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4e7 00000080:00010000:3.0:1788572410.613956:0:163384:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4e7 from it ffffaf7301dc7db0 00010000:00010000:3.0:1788572410.613957:0:163384:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572410.613961:0:163384:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572410.613966:0:163384:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572410.613969:0:163384:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=33188, excl=1 00000080:00200000:3.0:1788572410.613972:0:163384:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) mode 100644 dev 0 00010000:00010000:3.0:1788572410.613991:0:163384:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572410.614033:0:163384:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572410.614037:0:163384:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572410.614040:0:163384:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb4e7 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bebf1 expref: -99 pid: 163377 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:3.0:1788572410.614059:0:163384:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572410.614072:0:163384:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3e680 x1875444569535360/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572410.614096:0:163384:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163384:1875444569535360:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572410.614180:0:163384:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a79c0 going to sleep for 50 seconds 00000100:00100000:2.0:1788572410.621756:0:163384:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163384:1875444569535360:192.168.202.157@tcp:36:mcreate.0 00000080:00200000:3.0:1788572410.624343:0:163384:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x6:0x0](ffff920a50d380c8) 00000080:00200000:3.0:1788572410.624398:0:163384:0:(namei.c:2070:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, unhashed 1 00000080:00200000:2.0:1788572410.645285:0:163387:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572410.645854:0:163387:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572410.649737:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572410.649757:0:163387:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572410.649768:0:163387:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572410.649798:0:163387:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163387:1875444569535488:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572410.649906:0:163387:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e49c0 going to sleep for 50 seconds 00000100:00100000:3.0:1788572410.654957:0:163387:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163387:1875444569535488:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572410.654985:0:163387:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.654998:0:163387:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655011:0:163387:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655022:0:163387:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655035:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655044:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655053:0:163387:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572410.655065:0:163387:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac5500 x1875444569535488/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572410.655078:0:163387:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655092:0:163387:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572410.655098:0:163387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572410.655126:0:163387:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572410.655132:0:163387:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4ee 00000080:00010000:3.0:1788572410.655137:0:163387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.655140:0:163387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4ee from it ffffaf7301dafb58 00010000:00010000:3.0:1788572410.655144:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655153:0:163387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572410.655177:0:163387:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572410.655193:0:163387:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:3.0:1788572410.655198:0:163387:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572410.655218:0:163387:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572410.655255:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655265:0:163387:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.655274:0:163387:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 163387 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572410.655294:0:163387:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163387:1875444569535616:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572410.655459:0:163387:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7cc0 going to sleep for 50 seconds 00000100:00100000:3.0:1788572410.659114:0:163387:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163387:1875444569535616:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572410.659138:0:163387:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659158:0:163387:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x6:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659165:0:163387:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659175:0:163387:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659185:0:163387:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659194:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659206:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659214:0:163387:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572410.659224:0:163387:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a51b3f800 x1875444569535616/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572465 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572410.659235:0:163387:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659246:0:163387:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/6 (0 0) 00000080:00200000:3.0:1788572410.659252:0:163387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572410.659273:0:163387:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x6:0x0], updating i_size 0 00000080:00200000:3.0:1788572410.659276:0:163387:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x6:0x0](ffff920a50d380c8): rc = 0 00000080:00200000:3.0:1788572410.659293:0:163387:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x6:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572410.659298:0:163387:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x6:0x0](ffff920a50d380c8) for lock 0x8364c713e25bb4f5 00000080:00010000:3.0:1788572410.659324:0:163387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x6:0x0](ffff920a50d380c8) 00000080:00010000:3.0:1788572410.659328:0:163387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4f5 from it ffffaf7301dafb20 00010000:00010000:3.0:1788572410.659331:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572410.659340:0:163387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572410.659376:0:163387:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x200027501:0x6:0x0](ffff920a50d380c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572410.659383:0:163387:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x6:0x0](ffff920a50d380c8),name=fsa-oleg257-client.virtnet 00000002:00010000:3.0:1788572410.659404:0:163387:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x6:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.659413:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572410.659424:0:163387:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572410.659431:0:163387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.659435:0:163387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x6:0x0](ffff920a50d380c8) 00000080:00010000:3.0:1788572410.659439:0:163387:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x6:0x0](ffff920a50d380c8) for lock 0x8364c713e25bb4f5 00000080:00010000:3.0:1788572410.659442:0:163387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4f5 from it ffffaf7301dafcb8 00010000:00010000:3.0:1788572410.659445:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572410.659451:0:163387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572410.659504:0:163387:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572410.659512:0:163387:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.659531:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572410.659540:0:163387:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572410.659546:0:163387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.659551:0:163387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.659554:0:163387:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4ee 00000080:00010000:3.0:1788572410.659557:0:163387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4ee from it ffffaf7301dafc10 00010000:00010000:3.0:1788572410.659561:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572410.659568:0:163387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572410.659577:0:163387:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572410.662618:0:163387:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:3.0:1788572410.662751:0:163387:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572410.662782:0:163387:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572410.662802:0:163387:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572410.662818:0:163387:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572410.662826:0:163387:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572410.662835:0:163387:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572410.662838:0:163387:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb4ee 00000080:00010000:3.0:1788572410.662845:0:163387:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb4ee from it ffffaf7301dafd68 00010000:00010000:3.0:1788572410.662849:0:163387:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572410.662857:0:163387:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572410.662877:0:163387:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572410.662884:0:163387:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:3.0:1788572410.662913:0:163387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572410.662940:0:163387:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572410.662950:0:163387:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x6:0x0]: losing layout lock 00010000:00010000:3.0:1788572410.662992:0:163387:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572410.662999:0:163387:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572410.663005:0:163387:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572410.663008:0:163387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6800/0x8364c713e25bb4f5 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x6:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bec29 expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572410.663025:0:163387:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816600/0x8364c713e25bb4ee lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bec0d expref: -99 pid: 163387 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572410.663044:0:163387:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572410.663056:0:163387:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3f800 x1875444569535872/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1788572412.316583:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572412.895117:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569536000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.895209:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536256:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.895238:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536512:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.895269:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:2.0:1788572412.895276:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536128:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.895301:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536384:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.899482:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536128:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.899505:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536384:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.899516:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536256:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.899528:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569536512:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572412.912722:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569536000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572413.404663:0:163277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896600 going to sleep for 25 seconds 00010000:00010000:1.0:1788572414.364272:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572416.412251:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572418.012169:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569536640:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572418.012307:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569536896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572418.012337:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569537152:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572418.012371:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572418.012378:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569536768:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572418.012415:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569537024:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572418.021658:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569536768:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572418.021690:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569537024:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572418.021701:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569536640:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572418.021712:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569536896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572418.021724:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569537152:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572418.460425:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572418.553336:0:163463:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 00000100:00100000:1.0:1788572418.578204:0:163277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bc700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163277:1875444569537408:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572418.578257:0:163277:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48350000 x1875444569531520/t347892350988(347892350988) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572420.509273:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572421.542029:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572422.557283:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572423.133256:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569537536:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.133368:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537792:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.133397:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569538048:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.133424:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572423.133430:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.133450:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537920:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.139254:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bdc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.139273:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bce00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569537536:192.168.202.157@tcp:400:kworker.0 00000100:00080000:1.0:1788572423.139287:0:2397:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0001-mdc-ffff920a47b93800 of lustre-MDT0001_UUID@192.168.202.157@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788572423.139293:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788572423.139296:0:2397:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff920a47b93800: Connection to lustre-MDT0001 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788572423.139303:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572423.139307:0:2397:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9058 00000100:00080000:1.0:1788572423.139316:0:2397:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572423.139328:0:2397:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572423.139360:0:2397:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350000 x1875444569538176/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572423.139395:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bd500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537792:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.145284:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478be680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569537920:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572423.147649:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bdf80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569538048:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572423.591437:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572424.603954:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572425.630477:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572426.651849:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572427.678789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572428.252088:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569538304:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572428.252298:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572428.252308:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569538432:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572428.253302:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572428.253309:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572428.253314:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572428.253322:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9181 00000100:00080000:3.0:1788572428.253330:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572428.253337:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572428.253348:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572428.253372:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3d180 x1875444569538560/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572428.254122:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569538816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572428.254181:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572428.254187:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569538688:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572428.256712:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569538432:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572428.256728:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569538304:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572428.260791:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569538688:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572428.260815:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569538816:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572428.700052:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572429.724208:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572430.747859:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572431.773086:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572432.795857:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572433.376786:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572433.376794:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572433.376799:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572433.376812:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9186 00000100:00080000:3.0:1788572433.376823:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:3.0:1788572433.376832:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572433.376843:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572433.376859:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51b3df80 x1875444569539200/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572433.376894:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572433.376898:0:103578:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00080000:3.0:1788572433.376906:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572433.376908:0:103578:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00100000:3.0:1788572433.376937:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569539072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572433.377098:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569538944:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572433.396054:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569539072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572433.401525:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569538944:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572433.821382:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572435.869133:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572437.926617:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572438.496422:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569539584:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572438.498683:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bc380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569539584:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572438.499219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569539840:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572438.501111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572438.501122:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572438.501128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bea00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569539840:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572443.612782:0:124398:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572443.612790:0:124398:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572443.612795:0:124398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572443.612808:0:124398:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9197 00000100:00080000:2.0:1788572443.612821:0:124398:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572443.612830:0:124398:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572443.612845:0:124398:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572443.612868:0:124398:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7480 x1875444569540224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572443.612925:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569540096:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572443.613069:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1788572443.613076:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569539968:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572443.617103:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569539968:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572443.617134:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569540096:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572446.110568:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572447.300011:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572447.304549:0:160922:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:2.0:1788572447.304684:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584420736 00000100:00100000:2.0:1788572447.304703:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584420736 00000100:00100000:2.0:1788572447.304843:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 6 00000100:00100000:2.0:1788572447.304855:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a529be850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f79285ed-4685-45a8-8731-5e2522a143f3+3:196584:x1875444584420736:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572447.304873:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572447.304972:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572447.304979:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a529be850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f79285ed-4685-45a8-8731-5e2522a143f3+3:196584:x1875444584420736:12345-192.168.202.157@tcp:104: Request processed in 128us (4979us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572447.304987:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 6 00010000:00010000:1.0:1788572447.305067:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572447.305075:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572447.305084:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47817800 already unused, calling callback (ffffffffc1336c20) 10000000:00010000:1.0:1788572447.305088:0:160930:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572447.305098:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572447.305107:0:160930:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572447.305113:0:160930:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572447.305159:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572447.305166:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572447.305187:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572447.305195:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a47817800/0x8364c713e25bb3ac lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109be1c5 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572447.305213:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572447.305216:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572447.305219:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572447.305228:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572447.305243:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569540352:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572447.353330:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569540352:192.168.202.157@tcp:103:ldlm_bl.0 10000000:01000000:2.0:1788572448.413800:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788572448.413807:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 340 10000000:01000000:2.0:1788572448.413812:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572448.413856:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572448.413867:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572448.413880:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572448.413922:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569540480:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572448.414231:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572448.425037:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569540480:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572448.425070:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572448.425088:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572448.425170:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569540608:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788572448.425252:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:1.0:1788572448.433512:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569540608:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00080000:1.0:1788572448.433562:0:160932:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a51ac4800 already not connected (conn 6, was 0): DISCONN 00010000:00080000:1.0:1788572448.433576:0:160932:0:(ldlm_lib.c:85:import_set_conn()) imp ffff920a51ac4800@lustre-MDT0001-mdc-ffff920a47b93800: found existing conn 192.168.202.157@tcp, moved to head 00000100:00080000:1.0:1788572448.433581:0:160932:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572448.433614:0:160932:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:1.0:1788572448.433624:0:160932:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572448.433635:0:160932:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572448.433664:0:160932:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569540736/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:1.0:1788572448.433735:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:1.0:1788572448.433746:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572448.433756:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572448.433823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569540736:192.168.202.157@tcp:38:ll_cfg_requeue.0 00010000:00080000:1.0:1788572448.437064:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a48350a80 x1875444569540736/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572458 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572448.437087:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0001-mdc-ffff920a47b93800: connect to target with instance 6 00000100:00080000:1.0:1788572448.437108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572448.437115:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0001_UUID 00000100:00100000:1.0:1788572448.437118:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 21474837549 gen 1 00000100:00080000:1.0:1788572448.437132:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a51ac4800 from lustre-MDT0001_UUID committed 21474837549 last 0 00000100:00080000:1.0:1788572448.437136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from REPLAY to REPLAY_LOCKS 00000100:00100000:3.0:1788572448.437666:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569540736:192.168.202.157@tcp:38:ll_cfg_requeue.0 00010000:00080000:1.0:1788572448.437719:0:163704:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0001-mdc-ffff920a47b93800 to lustre-MDT0001_UUID@192.168.202.157@tcp 00010000:00010000:1.0:1788572448.437723:0:163704:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0001-mdc-ffff920a47b93800 (0) 00010000:00010000:1.0:1788572448.437728:0:163704:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0001-mdc-ffff920a47b93800 00000100:00080000:1.0:1788572448.438163:0:163704:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:00100000:3.0:1788572448.439437:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569540864:192.168.202.157@tcp:400:ldlm_lock_repla.0 00000100:00080000:3.0:1788572448.735658:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level REPLAY_WAIT/7 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572448.735665:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: REPLAY_WAIT) 00000100:00100000:3.0:1788572448.735691:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569541120:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572448.735789:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572448.735796:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569540992:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572448.751694:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569540992:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572448.751756:0:2400:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:3.0:1788572448.751765:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569541120:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572448.751868:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 0 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572448.751885:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572448.751886:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572448.751897:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572450.213187:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572452.251826:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00080000:1.0:1788572452.390690:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a48351c00 x1875444569540864/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572478 ref 1 fl Rpc:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572452.390717:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from REPLAY_WAIT to RECOVER 00000100:00080000:1.0:1788572452.390736:0:2396:0:(client.c:3248:ptlrpc_resend_req()) @@@ going to resend req@ffff920a48351c00 x1875444569540864/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572478 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572452.390745:0:2396:0:(client.c:3257:ptlrpc_resend_req()) @@@ it has reply, so skip it req@ffff920a48351c00 x1875444569540864/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572478 ref 1 fl Interpret:RQU/2c0/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572452.390752:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from RECOVER to FULL 00000100:02000000:3.0:1788572452.394628:0:2396:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) lustre-MDT0001-mdc-ffff920a47b93800: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00100000:3.0:1788572452.394645:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569540864:192.168.202.157@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572452.763945:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569541248:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572452.764064:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569541504:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572452.764102:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572452.764108:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569541376:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572452.767623:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569541376:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572452.767643:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569541248:192.168.202.157@tcp:400:kworker.0 00010000:00080000:2.0:1788572452.767656:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a60ac7800 x1875444569541504/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572482 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572452.767668:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569541504:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572454.301841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572455.771944:0:163782:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572455.771961:0:163782:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572455.771985:0:163782:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.772049:0:163782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 163782 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.772060:0:163782:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163782 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.772071:0:163782:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 163782 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572455.772098:0:163782:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541632:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572455.772244:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 25 seconds 00000100:00100000:2.0:1788572455.775022:0:163782:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541632:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572455.775049:0:163782:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775062:0:163782:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775074:0:163782:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775086:0:163782:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775100:0:163782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775108:0:163782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775116:0:163782:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572455.775129:0:163782:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac7800 x1875444569541632/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572485 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572455.775144:0:163782:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775157:0:163782:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572455.775164:0:163782:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572455.775188:0:163782:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:2.0:1788572455.775193:0:163782:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572455.775199:0:163782:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572455.775202:0:163782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300d17cb8 00010000:00010000:2.0:1788572455.775205:0:163782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572455.775213:0:163782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572455.775866:0:163782:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572455.775872:0:163782:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572455.775888:0:163782:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.775899:0:163782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.775912:0:163782:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572455.775919:0:163782:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.775924:0:163782:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572455.775927:0:163782:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572455.775931:0:163782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300d17cb8 00010000:00010000:2.0:1788572455.775935:0:163782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572455.775943:0:163782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572455.776035:0:163782:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572455.776042:0:163782:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572455.776049:0:163782:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.776058:0:163782:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572455.776065:0:163782:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572455.776068:0:163782:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572455.776071:0:163782:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572455.776075:0:163782:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300d17bb0 00010000:00010000:2.0:1788572455.776078:0:163782:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572455.776085:0:163782:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572455.776093:0:163782:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572455.776099:0:163782:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:2.0:1788572455.776161:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.776173:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00800000:00000004:2.0:1788572455.776180:0:163782:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9169, max_age 9213 00000100:00100000:2.0:1788572455.776212:0:163782:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541760:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572455.776275:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.782938:0:163782:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541760:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572455.782970:0:163782:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023636/1024000 00000080:00200080:2.0:1788572455.783050:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.783067:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00800000:00000004:2.0:1788572455.783075:0:163782:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9169, max_age 9213 00000100:00100000:2.0:1788572455.783113:0:163782:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541888:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572455.783187:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 25 seconds 00010000:00080000:2.0:1788572455.791366:0:163782:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a60ac7800 x1875444569541888/t0(0) o41->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572485 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572455.791388:0:163782:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569541888:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572455.791416:0:163782:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317330/353529 objects 1023723/1024000 00000080:00200080:2.0:1788572455.791492:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791512:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791520:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791523:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791527:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791530:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791534:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791537:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791541:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791545:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791548:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791551:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791554:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791556:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791561:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791564:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791568:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791572:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791576:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791580:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791600:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791606:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791611:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791615:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791619:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791623:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791627:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791630:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791634:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791638:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791643:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791646:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791650:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791653:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791657:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791660:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791665:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791667:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791671:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791675:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791679:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791683:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791687:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791691:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791695:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791699:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791703:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791707:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791711:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791714:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791718:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791721:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791725:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791728:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791732:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791735:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791749:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791754:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791758:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791761:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791765:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791768:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791772:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791774:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791779:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791782:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791786:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791790:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791794:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791797:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791801:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791804:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791808:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791811:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791815:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791819:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791823:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791827:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791832:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791835:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791840:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791843:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791847:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791851:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791855:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791858:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791862:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791866:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791870:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791875:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791879:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791883:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791888:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791891:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791895:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791898:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791903:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791906:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791910:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791914:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791918:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791921:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791925:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791928:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791932:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791936:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791940:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791943:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791946:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791949:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791953:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791956:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791960:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791963:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791966:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791969:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791973:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791976:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791980:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791983:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791987:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791991:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.791995:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.791998:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.792001:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00800000:00000080:2.0:1788572455.792004:0:163782:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.792008:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.792014:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00020000:00000004:2.0:1788572455.792022:0:163782:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9169, max_age 9213 00000100:00080000:2.0:1788572455.792038:0:163782:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9139 00000100:00080000:2.0:1788572455.792053:0:163782:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572455.792067:0:163782:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788572455.792094:0:163782:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5f80 x1875444569542016/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572455.792148:0:163782:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a60ac7800 x1875444569542144/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572455.792157:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 0 seconds 00000100:00100000:1.0:1788572455.792280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569542016:192.168.202.157@tcp:8:lfs.0 00000100:00100000:1.0:1788572455.795066:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a47b93800: committing for last_committed 38654705666 gen 3 00000100:00080000:1.0:1788572455.795083:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff920a47b93800: connect to target with instance 9 00000100:00080000:1.0:1788572455.795093:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572455.795107:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:1.0:1788572455.795110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572455.795121:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572455.795127:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572455.795142:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000e08c2b2d): req@ffff920a60ac7800 x1875444569542144/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572455.795170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569542016:192.168.202.157@tcp:8:lfs.0 00000100:00100000:2.0:1788572455.795326:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.797852:0:163782:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569542144:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572455.797883:0:163782:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:2.0:1788572455.797972:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.797992:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00020000:00000004:2.0:1788572455.798000:0:163782:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9169, max_age 9213 00000100:00080000:2.0:1788572455.798011:0:163782:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9139 00000100:00080000:2.0:1788572455.798020:0:163782:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572455.798030:0:163782:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788572455.798055:0:163782:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569542272/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572455.798098:0:163782:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a60ac7800 x1875444569542400/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572455.798107:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 0 seconds 00000100:00100000:1.0:1788572455.798237:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569542272:192.168.202.157@tcp:8:lfs.0 00000100:00100000:1.0:1788572455.801212:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff920a47b93800: committing for last_committed 0 gen 3 00000100:00080000:1.0:1788572455.801229:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff920a47b93800: connect to target with instance 1 00000100:00080000:1.0:1788572455.801239:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:1.0:1788572455.801249:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:1.0:1788572455.801253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:1.0:1788572455.801262:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:1.0:1788572455.801267:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:1.0:1788572455.801281:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000e08c2b2d): req@ffff920a60ac7800 x1875444569542400/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572455.801304:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569542272:192.168.202.157@tcp:8:lfs.0 00000100:00100000:2.0:1788572455.801558:0:163782:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572455.805368:0:163782:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:163782:1875444569542400:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572455.805396:0:163782:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:2.0:1788572455.805475:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805493:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805500:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805503:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805507:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805510:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805513:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805517:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805520:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805525:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805529:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805532:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805536:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805540:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805544:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805547:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805551:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805554:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805558:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805562:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805566:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805569:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805572:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805576:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805580:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805583:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805629:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805634:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805638:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805642:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805646:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805649:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805653:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805656:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805660:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805664:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805668:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805672:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805676:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805680:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805685:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805688:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805693:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805696:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805700:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805703:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805707:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805711:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805715:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805719:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805731:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805735:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805750:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805754:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805759:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805762:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805766:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805770:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805775:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805779:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805783:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805787:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805790:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805794:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805798:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805802:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805806:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805809:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805813:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805817:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805821:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805825:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805828:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805831:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805835:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805838:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805842:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805845:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805849:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805852:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805857:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805860:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805863:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805867:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805871:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805874:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805877:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805880:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805884:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805887:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805890:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805893:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805896:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805899:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805903:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805905:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805908:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805911:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805915:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805917:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.805920:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.805923:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806036:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806043:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806049:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806053:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806057:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806060:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806064:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806068:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806072:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806075:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806079:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806082:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806086:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806089:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806092:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806095:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806099:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806102:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806105:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806108:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806111:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806114:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200080:2.0:1788572455.806118:0:163782:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff64bae9a0 00020000:00000080:2.0:1788572455.806120:0:163782:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45852400 uarg=0000000000000000 00000080:00200008:2.0:1788572455.806132:0:163782:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:1.0:1788572456.348095:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572457.951261:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569542528:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.951395:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542784:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.951427:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543040:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.951457:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572457.951464:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.951489:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542912:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.961253:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.961274:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542912:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.961284:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569542528:192.168.202.157@tcp:400:kworker.0 00010000:00080000:2.0:1788572457.961296:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a51b3e300 x1875444569542784/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572473 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572457.961307:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569542784:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572457.961318:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543040:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572458.415742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572460.449708:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572461.480137:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572461.798108:0:164063:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 00010000:00010000:1.0:1788572462.495957:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572463.076775:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543296:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.076871:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543552:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.076905:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572463.076910:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569543168:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.076937:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543424:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.076959:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543680:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.079809:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569543168:192.168.202.157@tcp:400:kworker.0 00010000:00080000:2.0:1788572463.079831:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a60ac5180 x1875444569543424/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572479 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572463.079843:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543424:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.079857:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543296:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.081336:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543680:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572463.081361:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569543552:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572464.540055:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572466.027399:0:164249:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572466.027416:0:164249:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.027441:0:164249:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.027463:0:164249:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572466.027479:0:164249:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572466.027488:0:164249:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.027496:0:164249:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.027500:0:164249:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.027506:0:164249:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300c97cb8 00010000:00010000:1.0:1788572466.027510:0:164249:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572466.027519:0:164249:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788572466.028556:0:164249:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572466.028564:0:164249:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.028580:0:164249:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.028742:0:164249:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572466.028756:0:164249:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572466.028764:0:164249:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.028771:0:164249:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.028774:0:164249:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.028781:0:164249:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300c97cb8 00010000:00010000:1.0:1788572466.028784:0:164249:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572466.028791:0:164249:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572466.028992:0:164249:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.029001:0:164249:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.029008:0:164249:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572466.029015:0:164249:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572466.029021:0:164249:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.029025:0:164249:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.029027:0:164249:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.029031:0:164249:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300c97bb0 00010000:00010000:1.0:1788572466.029034:0:164249:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572466.029040:0:164249:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572466.029049:0:164249:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572466.029056:0:164249:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:1.0:1788572466.033236:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.033266:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00800000:00000004:1.0:1788572466.033276:0:164249:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9214, max_age 9223 00000100:00100000:1.0:1788572466.033320:0:164249:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569543808:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572466.033399:0:164249:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.037467:0:164249:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569543808:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572466.037496:0:164249:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317230/353529 objects 1023636/1024000 00000080:00200080:1.0:1788572466.037577:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.037609:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00800000:00000004:1.0:1788572466.037617:0:164249:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9214, max_age 9223 00000100:00100000:1.0:1788572466.037649:0:164249:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569543936:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572466.037716:0:164249:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6b40 going to sleep for 11 seconds 00010000:00080000:1.0:1788572466.042927:0:164249:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a48350700 x1875444569543936/t0(0) o41->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572482 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572466.042950:0:164249:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569543936:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572466.042978:0:164249:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317330/353529 objects 1023723/1024000 00000080:00200080:1.0:1788572466.043105:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043123:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043131:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043135:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043139:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043143:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043146:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043149:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043153:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043156:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043161:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043164:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043168:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043172:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043176:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043180:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043184:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043188:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043192:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043196:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043200:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043204:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043208:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043212:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043216:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043219:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043223:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043227:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043231:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043235:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043239:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043242:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043246:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043249:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043253:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043256:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043259:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043263:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043266:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043269:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043273:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043276:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043280:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043283:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043286:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043290:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043294:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043297:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043300:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043303:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043307:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043310:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043314:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043318:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043323:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043326:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043331:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043334:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043338:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043341:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043345:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043348:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043353:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043367:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043371:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043375:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043378:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043381:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043385:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043388:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043392:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043396:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043400:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043403:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043407:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043410:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043414:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043418:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043422:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043426:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043429:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043433:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043436:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043440:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043443:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043447:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043451:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043454:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043457:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043461:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043465:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043469:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043473:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043476:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043481:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043484:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043488:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043492:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043496:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043499:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043503:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043506:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043511:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043514:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043518:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043522:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043525:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043529:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043533:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043537:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043541:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043544:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043549:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043552:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043556:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043560:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043564:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043567:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043571:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043574:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043578:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043581:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043617:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043622:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043626:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00800000:00000080:1.0:1788572466.043629:0:164249:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.043634:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.043639:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00020000:00000004:1.0:1788572466.043647:0:164249:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9214, max_age 9223 00000100:00100000:1.0:1788572466.043681:0:164249:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569544064:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572466.043750:0:164249:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.046148:0:164249:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569544064:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572466.046175:0:164249:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:1.0:1788572466.046227:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.046243:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00020000:00000004:1.0:1788572466.046250:0:164249:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9214, max_age 9223 00000100:00100000:1.0:1788572466.046278:0:164249:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569544192:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572466.046338:0:164249:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.048768:0:164249:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164249:1875444569544192:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572466.048880:0:164249:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:1.0:1788572466.048938:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.048956:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.048963:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.048967:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.048972:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.048976:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.048980:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049053:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049061:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049066:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049069:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049072:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049075:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049079:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049083:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049087:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049091:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049094:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049097:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049100:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049103:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049106:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049110:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049113:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049117:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049120:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049124:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049127:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049131:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049134:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049138:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049141:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049145:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049148:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049151:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049155:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049158:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049161:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049165:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049168:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049172:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049462:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049470:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049474:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049477:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049480:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049483:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049486:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049490:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049493:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049496:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049500:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049504:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049507:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049511:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049515:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049518:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049522:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049526:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049529:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049533:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049536:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049540:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049544:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049547:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049551:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049555:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049558:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049562:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049565:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049569:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049573:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049576:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049580:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049584:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049887:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049895:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049898:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049902:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049906:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049910:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049914:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049918:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049921:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049925:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049928:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049932:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049935:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049940:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049943:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049947:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049951:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049955:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049958:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049962:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049966:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049970:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049974:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.049978:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.049982:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050197:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050203:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050207:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050210:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050214:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050217:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050221:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050225:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050228:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050231:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050235:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050239:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050242:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050247:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050250:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050253:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050257:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050260:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050264:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050267:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050271:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050275:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050279:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050282:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200080:1.0:1788572466.050286:0:164249:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdecb473b0 00020000:00000080:1.0:1788572466.050289:0:164249:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b000 uarg=0000000000000000 00000080:00200008:1.0:1788572466.050303:0:164249:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:3.0:1788572466.205230:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572466.205260:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572466.205945:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572466.205969:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00000080:00200000:3.0:1788572466.205978:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572466.205988:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572466.205991:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:3.0:1788572466.205999:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:3.0:1788572466.206002:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 1 00010000:00010000:3.0:1788572466.206010:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572466.206026:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.206033:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:3.0:1788572466.206038:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.206042:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:3.0:1788572466.206045:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.206057:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.206062:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-0, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a46289980, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788572466.206069:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-0, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:3.0:1788572466.206102:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-0,[0x240002729:0x17:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788572466.206159:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.206166:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572466.206179:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572466.206193:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ea00 x1875444569545216/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572466.206201:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572466.206225:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ea00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569545216:192.168.202.157@tcp:101:createmany.0 00000100:00100000:3.0:1788572466.206555:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865240 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.253853:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bc000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569545984:192.168.202.157@tcp:35:createmany.0 00000002:00100000:3.0:1788572466.259154:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a478bca80 x1875444569545728/t25769803781(25769803781) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572466.259262:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572466.259602:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572466.259623:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572466.259639:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572466.259648:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572466.259654:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572466.259658:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:3.0:1788572466.259665:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:3.0:1788572466.259670:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572466.259685:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572466.259701:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.259708:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:3.0:1788572466.259713:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.259717:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:3.0:1788572466.259721:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.259734:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.259739:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-2, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a51dc2a80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788572466.259747:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-2, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:3.0:1788572466.259767:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-2,[0x240002729:0x19:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788572466.259832:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.259841:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572466.259854:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572466.259862:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3f100 x1875444569546240/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572466.259871:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572466.259892:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569546240:192.168.202.157@tcp:101:createmany.0 00000100:00100000:3.0:1788572466.260042:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865a80 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.271708:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569546240:192.168.202.157@tcp:101:createmany.0 00010000:00010000:3.0:1788572466.271735:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271766:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x19:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271776:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271788:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271813:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271828:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271838:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271847:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572466.271860:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51b3f100 x1875444569546240/t25769803783(25769803783) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572466.271878:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51b3f100 x1875444569546240/t25769803783(25769803783) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572466.271891:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572466.271904:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.271914:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/25 (0 0) 00000080:00200000:3.0:1788572466.271921:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572466.271934:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-2, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:3.0:1788572466.271964:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x19:0x0](ffff920a50d3f448) 00000080:00200000:3.0:1788572466.272044:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x19:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572466.272050:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x19:0x0](ffff920a50d3f448) 00000080:00010000:3.0:1788572466.272054:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x19:0x0](ffff920a50d3f448) for lock 0x8364c713e25bb526 00000080:00010000:3.0:1788572466.272065:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x19:0x0](ffff920a50d3f448) 00000080:00200008:3.0:1788572466.272078:0:164272:0:(file.c:1090:ll_file_open()) START file f-2:[0x240002729:0x19:0x0](ffff920a50d3f448), flags 100102 00000002:00100000:3.0:1788572466.272101:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51b3f100 x1875444569546240/t25769803783(25769803783) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572466.272142:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb526 from it ffff920a51b4e960 00010000:00010000:3.0:1788572466.272147:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.272157:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb526 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x19:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bef6a expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.272182:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x19:0x0](ffff920a50d3f448), inode mode 81a4 mask 40 00000080:00200008:3.0:1788572466.272206:0:164272:0:(file.c:479:ll_file_release()) START file f-2:[0x240002729:0x19:0x0](ffff920a50d3f448), flags 100002 00000002:00100000:3.0:1788572466.272250:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51b3f100 x1875444569546240/t25769803783(25769803783) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572466.272260:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572466.272267:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3df80 x1875444569546496/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572466.272290:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3df80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569546496:192.168.202.157@tcp:35:createmany.0 00000100:00100000:3.0:1788572466.272358:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a518650c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.317943:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bea00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547008:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572466.317982:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a478bfb80 x1875444569546752/t25769803785(25769803785) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572466.318070:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.318092:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.318117:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572466.318133:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:1.0:1788572466.318142:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.318149:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.318153:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.318162:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:1.0:1788572466.318180:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:1.0:1788572466.318190:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572466.318216:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.318223:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:1.0:1788572466.318229:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.318233:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572466.318237:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.318253:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572466.318258:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-4, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a52cf2d80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572466.318268:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-4, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572466.318302:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-4,[0x240002729:0x1b:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572466.318362:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.318371:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572466.318383:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572466.318390:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350700 x1875444569547264/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572466.318399:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572466.318638:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547264:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572466.318780:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6b40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.330112:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547264:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572466.330146:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330194:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330205:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330217:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330230:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330245:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330254:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330262:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572466.330281:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48350700 x1875444569547264/t25769803787(25769803787) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572466.330306:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350700 x1875444569547264/t25769803787(25769803787) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572466.330321:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572466.330334:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330345:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/27 (0 0) 00000080:00200000:1.0:1788572466.330352:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572466.330366:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-4, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572466.330402:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x1b:0x0](ffff920a50d4b248) 00000080:00200000:1.0:1788572466.330842:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x1b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572466.330849:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x1b:0x0](ffff920a50d4b248) 00000080:00010000:1.0:1788572466.330853:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1b:0x0](ffff920a50d4b248) for lock 0x8364c713e25bb534 00000080:00010000:1.0:1788572466.330863:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1b:0x0](ffff920a50d4b248) 00000080:00200008:1.0:1788572466.330876:0:164272:0:(file.c:1090:ll_file_open()) START file f-4:[0x240002729:0x1b:0x0](ffff920a50d4b248), flags 100102 00000002:00100000:1.0:1788572466.330896:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350700 x1875444569547264/t25769803787(25769803787) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572466.330930:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb534 from it ffff920a48138cc0 00010000:00010000:1.0:1788572466.330935:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.330949:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572466.330968:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1b:0x0](ffff920a50d4b248), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572466.330994:0:164272:0:(file.c:479:ll_file_release()) START file f-4:[0x240002729:0x1b:0x0](ffff920a50d4b248), flags 100002 00000002:00100000:1.0:1788572466.331036:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48350700 x1875444569547264/t25769803787(25769803787) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572466.331046:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572466.331051:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353800 x1875444569547520/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572466.331073:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547520:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572466.331155:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.336840:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547520:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572466.336884:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48350700 x1875444569547264/t25769803787(25769803787) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572466.336991:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.337014:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.337043:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572466.337059:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:1.0:1788572466.337068:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.337075:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.337079:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.337086:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:1.0:1788572466.337090:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1788572466.337099:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788572466.337119:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.337126:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:1.0:1788572466.337132:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.337136:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572466.337140:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.337155:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572466.337158:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-5, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a52077800, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572466.337239:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-5, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572466.337259:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-5,[0x240002729:0x1c:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572466.337388:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.337398:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572466.337410:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572466.337486:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353100 x1875444569547776/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572466.337497:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572466.337519:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547776:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572466.337846:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.347158:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569547776:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572466.347199:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347234:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347242:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347254:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347266:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347278:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347295:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347302:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572466.347319:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48353100 x1875444569547776/t25769803789(25769803789) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572466.347333:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48353100 x1875444569547776/t25769803789(25769803789) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572466.347345:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572466.347358:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347369:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/28 (0 0) 00000080:00200000:1.0:1788572466.347375:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572466.347391:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-5, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572466.347689:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x1c:0x0](ffff920a50d4cb08) 00000080:00200000:1.0:1788572466.347776:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x1c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572466.347782:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x1c:0x0](ffff920a50d4cb08) 00000080:00010000:1.0:1788572466.347786:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1c:0x0](ffff920a50d4cb08) for lock 0x8364c713e25bb53b 00000080:00010000:1.0:1788572466.347807:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1c:0x0](ffff920a50d4cb08) 00000080:00200008:1.0:1788572466.347824:0:164272:0:(file.c:1090:ll_file_open()) START file f-5:[0x240002729:0x1c:0x0](ffff920a50d4cb08), flags 100102 00000002:00100000:1.0:1788572466.347848:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48353100 x1875444569547776/t25769803789(25769803789) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572466.347884:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb53b from it ffff920a48138cc0 00010000:00010000:1.0:1788572466.347889:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.347905:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb53b lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109befd3 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572466.347924:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1c:0x0](ffff920a50d4cb08), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572466.347952:0:164272:0:(file.c:479:ll_file_release()) START file f-5:[0x240002729:0x1c:0x0](ffff920a50d4cb08), flags 100002 00000002:00100000:1.0:1788572466.347992:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48353100 x1875444569547776/t25769803789(25769803789) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572466.348002:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572466.348007:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351f80 x1875444569548032/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572466.348027:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569548032:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572466.348113:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.353514:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569548032:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572466.353552:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48353100 x1875444569547776/t25769803789(25769803789) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572466.353651:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.353667:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.353683:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788572466.353697:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:1.0:1788572466.353705:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.353711:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.353714:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.353721:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:1.0:1788572466.353726:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:1.0:1788572466.353736:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:1.0:1788572466.353750:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.353756:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:1.0:1788572466.353761:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.353763:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572466.353766:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.353775:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572466.353779:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-6, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a52077c80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572466.353785:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-6, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572466.353814:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-6,[0x240002729:0x1d:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572466.353866:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.353873:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572466.353881:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572466.353886:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351c00 x1875444569548288/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572466.353892:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572466.353907:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569548288:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572466.354028:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572466.365275:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569548288:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572466.365443:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372788:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372819:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372830:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372844:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372865:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372874:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372881:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.372898:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48351c00 x1875444569548288/t25769803791(25769803791) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.372911:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351c00 x1875444569548288/t25769803791(25769803791) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.372926:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.372940:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.372951:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/29 (0 0) 00000080:00200000:2.0:1788572466.372958:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.372973:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-6, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.373010:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x1d:0x0](ffff920a50d29148) 00000080:00200000:2.0:1788572466.373169:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x1d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.373175:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x1d:0x0](ffff920a50d29148) 00000080:00010000:2.0:1788572466.373180:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x1d:0x0](ffff920a50d29148) for lock 0x8364c713e25bb542 00000080:00010000:2.0:1788572466.373191:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x1d:0x0](ffff920a50d29148) 00000080:00200008:2.0:1788572466.373205:0:164272:0:(file.c:1090:ll_file_open()) START file f-6:[0x240002729:0x1d:0x0](ffff920a50d29148), flags 100102 00000002:00100000:2.0:1788572466.373228:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351c00 x1875444569548288/t25769803791(25769803791) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.373277:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb542 from it ffff920a48138cc0 00010000:00010000:2.0:1788572466.373568:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.373580:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb542 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109beff6 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.373654:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x1d:0x0](ffff920a50d29148), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.373693:0:164272:0:(file.c:479:ll_file_release()) START file f-6:[0x240002729:0x1d:0x0](ffff920a50d29148), flags 100002 00000002:00100000:2.0:1788572466.373743:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48351c00 x1875444569548288/t25769803791(25769803791) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.373755:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.373761:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac4a80 x1875444569548544/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.373781:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569548544:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.373885:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d9c0 going to sleep for 11 seconds 00000080:00200000:2.0:1788572466.427530:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x20:0x0](ffff920a50d2ec08) 00000080:00200000:2.0:1788572466.428489:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x20:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.428501:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x20:0x0](ffff920a50d2ec08) 00000080:00010000:2.0:1788572466.428505:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x20:0x0](ffff920a50d2ec08) for lock 0x8364c713e25bb557 00000080:00010000:2.0:1788572466.428518:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x20:0x0](ffff920a50d2ec08) 00000080:00200008:2.0:1788572466.428533:0:164272:0:(file.c:1090:ll_file_open()) START file f-9:[0x240002729:0x20:0x0](ffff920a50d2ec08), flags 100102 00000002:00100000:2.0:1788572466.428565:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a478bce00 x1875444569549824/t25769803797(25769803797) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.428794:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb557 from it ffff920a52f779c0 00010000:00010000:2.0:1788572466.428812:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb557 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf05f expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.428825:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb557 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf05f expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.428855:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x20:0x0](ffff920a50d2ec08), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.429020:0:164272:0:(file.c:479:ll_file_release()) START file f-9:[0x240002729:0x20:0x0](ffff920a50d2ec08), flags 100002 00000002:00100000:2.0:1788572466.429071:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a478bce00 x1875444569549824/t25769803797(25769803797) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.429083:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.429088:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6a00 x1875444569550080/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.429110:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550080:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.429193:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.433019:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550080:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.433060:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a478bce00 x1875444569549824/t25769803797(25769803797) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.433144:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.433163:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.433181:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572466.433197:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:2.0:1788572466.433205:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.433210:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.433215:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.433222:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.433227:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:2.0:1788572466.433235:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572466.433250:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.433258:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.433263:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.433268:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.433271:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.433298:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.433303:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-10, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.433312:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-10, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.433333:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-10,[0x240002729:0x21:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.434102:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.434114:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.434125:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.434134:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6300 x1875444569550336/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.434143:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.434163:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550336:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.434313:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d840 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.442508:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550336:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.442535:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442567:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/0,1 mode: --/PR res: [0x240002729:0x21:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442576:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/0,1 mode: --/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442639:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442651:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442666:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442674:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442682:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.442694:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac6300 x1875444569550336/t25769803799(25769803799) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.442707:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac6300 x1875444569550336/t25769803799(25769803799) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.442722:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.442735:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442745:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/33 (0 0) 00000080:00200000:2.0:1788572466.442752:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.442766:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-10, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.442791:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x21:0x0](ffff920a50d2f448) 00000080:00200000:2.0:1788572466.442891:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x21:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.442897:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x21:0x0](ffff920a50d2f448) 00000080:00010000:2.0:1788572466.442900:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x21:0x0](ffff920a50d2f448) for lock 0x8364c713e25bb55e 00000080:00010000:2.0:1788572466.442911:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x21:0x0](ffff920a50d2f448) 00000080:00200008:2.0:1788572466.442927:0:164272:0:(file.c:1090:ll_file_open()) START file f-10:[0x240002729:0x21:0x0](ffff920a50d2f448), flags 100102 00000002:00100000:2.0:1788572466.442948:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac6300 x1875444569550336/t25769803799(25769803799) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.442984:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb55e from it ffff920a479e9f60 00010000:00010000:2.0:1788572466.442989:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.442998:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb55e lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x21:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf082 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.443025:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x21:0x0](ffff920a50d2f448), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.443053:0:164272:0:(file.c:479:ll_file_release()) START file f-10:[0x240002729:0x21:0x0](ffff920a50d2f448), flags 100002 00000002:00100000:2.0:1788572466.443096:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac6300 x1875444569550336/t25769803799(25769803799) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.443106:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.443112:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5180 x1875444569550592/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.443132:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550592:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.443203:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8df00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.445476:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550592:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.445508:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac6300 x1875444569550336/t25769803799(25769803799) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.445583:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.445640:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.445657:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572466.445670:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:2.0:1788572466.445678:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.445684:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.445687:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.445693:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.445697:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:2.0:1788572466.445706:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572466.445720:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.445727:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.445732:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.445736:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.445739:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.445751:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.445756:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-11, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.445764:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-11, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.445783:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-11,[0x240002729:0x22:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.445843:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.445851:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.445861:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.445867:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac7800 x1875444569550848/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.445888:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.445908:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550848:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.446035:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8df00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.454501:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569550848:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.454528:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454559:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x22:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454568:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454580:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454656:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454670:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454679:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454687:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.454699:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac7800 x1875444569550848/t25769803801(25769803801) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.454712:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac7800 x1875444569550848/t25769803801(25769803801) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.454726:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.454741:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.454754:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/34 (0 0) 00000080:00200000:2.0:1788572466.454761:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.454774:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-11, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.454818:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x22:0x0](ffff920a50d2cb08) 00000080:00200000:2.0:1788572466.455053:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x22:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.455058:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x22:0x0](ffff920a50d2cb08) 00000080:00010000:2.0:1788572466.455062:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x22:0x0](ffff920a50d2cb08) for lock 0x8364c713e25bb565 00000080:00010000:2.0:1788572466.455071:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x22:0x0](ffff920a50d2cb08) 00000080:00200008:2.0:1788572466.455210:0:164272:0:(file.c:1090:ll_file_open()) START file f-11:[0x240002729:0x22:0x0](ffff920a50d2cb08), flags 100102 00000002:00100000:2.0:1788572466.455234:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac7800 x1875444569550848/t25769803801(25769803801) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.455272:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb565 from it ffff920a479e9f60 00010000:00010000:2.0:1788572466.455277:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.455568:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817600/0x8364c713e25bb565 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x22:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf0a5 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.455626:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x22:0x0](ffff920a50d2cb08), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.455655:0:164272:0:(file.c:479:ll_file_release()) START file f-11:[0x240002729:0x22:0x0](ffff920a50d2cb08), flags 100002 00000002:00100000:2.0:1788572466.455701:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac7800 x1875444569550848/t25769803801(25769803801) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.455712:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.455718:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac7b80 x1875444569551104/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.455740:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551104:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.455828:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.459795:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551104:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.459845:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac7800 x1875444569550848/t25769803801(25769803801) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.459934:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.459951:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.459970:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788572466.459984:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:2.0:1788572466.459992:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.459998:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.460002:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.460009:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.460014:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:2.0:1788572466.460022:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788572466.460037:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.460044:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.460050:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.460054:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.460057:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.460071:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.460076:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-12, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.460084:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-12, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.460104:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-12,[0x240002729:0x23:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.460153:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.460162:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.460173:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.460179:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac4700 x1875444569551360/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.460189:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.460206:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551360:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.460342:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d0c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.464321:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551360:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.464355:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464386:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/0,1 mode: --/PR res: [0x240002729:0x23:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464395:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/0,1 mode: --/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464407:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464419:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464432:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464440:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464448:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.464461:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac4700 x1875444569551360/t25769803803(25769803803) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.464475:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4700 x1875444569551360/t25769803803(25769803803) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.464488:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.464501:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464513:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/35 (0 0) 00000080:00200000:2.0:1788572466.464520:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.464534:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-12, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.464568:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x23:0x0](ffff920a50d2e3c8) 00000080:00200000:2.0:1788572466.464729:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x23:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.464734:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x23:0x0](ffff920a50d2e3c8) 00000080:00010000:2.0:1788572466.464738:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x23:0x0](ffff920a50d2e3c8) for lock 0x8364c713e25bb56c 00000080:00010000:2.0:1788572466.464748:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x23:0x0](ffff920a50d2e3c8) 00000080:00200008:2.0:1788572466.464764:0:164272:0:(file.c:1090:ll_file_open()) START file f-12:[0x240002729:0x23:0x0](ffff920a50d2e3c8), flags 100102 00000002:00100000:2.0:1788572466.464785:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4700 x1875444569551360/t25769803803(25769803803) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.464834:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb56c from it ffff920a479e9f60 00010000:00010000:2.0:1788572466.464839:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.464849:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb56c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x23:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf0c8 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.464889:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x23:0x0](ffff920a50d2e3c8), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.464918:0:164272:0:(file.c:479:ll_file_release()) START file f-12:[0x240002729:0x23:0x0](ffff920a50d2e3c8), flags 100002 00000002:00100000:2.0:1788572466.464977:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac4700 x1875444569551360/t25769803803(25769803803) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.464989:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.464995:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac4000 x1875444569551616/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.465015:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551616:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.465102:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.470423:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551616:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.470469:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac4700 x1875444569551360/t25769803803(25769803803) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.470556:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.470575:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.470626:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572466.470643:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:2.0:1788572466.470653:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.470659:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.470663:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.470670:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.470675:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:2.0:1788572466.470685:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788572466.470703:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.470709:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.470715:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.470719:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.470722:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.470734:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.470739:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-13, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.470748:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-13, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.470767:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-13,[0x240002729:0x24:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.470830:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.470840:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.470850:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.470857:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5f80 x1875444569551872/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.470865:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.471014:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551872:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.471158:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8da80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.481995:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569551872:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.482023:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482056:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x24:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482066:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482077:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482087:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482101:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482109:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482116:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.482128:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac5f80 x1875444569551872/t25769803805(25769803805) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.482143:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac5f80 x1875444569551872/t25769803805(25769803805) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.482158:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.482173:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.482184:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/36 (0 0) 00000080:00200000:2.0:1788572466.482191:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.482205:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-13, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.482243:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x24:0x0](ffff920a50d280c8) 00000080:00200000:2.0:1788572466.482314:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x24:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.482319:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x24:0x0](ffff920a50d280c8) 00000080:00010000:2.0:1788572466.482323:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x24:0x0](ffff920a50d280c8) for lock 0x8364c713e25bb573 00000080:00010000:2.0:1788572466.482333:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x24:0x0](ffff920a50d280c8) 00000080:00200008:2.0:1788572466.483108:0:164272:0:(file.c:1090:ll_file_open()) START file f-13:[0x240002729:0x24:0x0](ffff920a50d280c8), flags 100102 00000002:00100000:3.0:1788572466.483225:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac5f80 x1875444569551872/t25769803805(25769803805) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572466.483266:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb573 from it ffff920a479e9f60 00010000:00010000:3.0:1788572466.483273:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.483284:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb573 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x24:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf0eb expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.483308:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x24:0x0](ffff920a50d280c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788572466.483344:0:164272:0:(file.c:479:ll_file_release()) START file f-13:[0x240002729:0x24:0x0](ffff920a50d280c8), flags 100002 00000002:00100000:3.0:1788572466.483403:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac5f80 x1875444569551872/t25769803805(25769803805) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572466.483413:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572466.483418:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3fb80 x1875444569552128/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572466.483442:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552128:192.168.202.157@tcp:35:createmany.0 00000100:00100000:3.0:1788572466.483530:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a518653c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.493103:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3fb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552128:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.493139:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac5f80 x1875444569551872/t25769803805(25769803805) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.493220:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.493239:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.493257:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788572466.493277:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:2.0:1788572466.493284:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.493292:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.493295:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.493302:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.493306:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:2.0:1788572466.493314:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1788572466.493330:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.493336:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.493342:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.493346:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.493349:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.493361:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.493366:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-14, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.493374:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-14, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.493393:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-14,[0x240002729:0x25:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.493442:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.493451:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.493463:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.493469:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac4e00 x1875444569552384/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.493480:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.493501:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552384:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.493663:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d780 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.509150:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552384:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.509177:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509208:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/0,1 mode: --/PR res: [0x240002729:0x25:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509216:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/0,1 mode: --/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509228:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509240:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509254:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509269:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509277:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.509289:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac4e00 x1875444569552384/t25769803807(25769803807) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.509303:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4e00 x1875444569552384/t25769803807(25769803807) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.509316:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.509329:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509362:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/37 (0 0) 00000080:00200000:2.0:1788572466.509368:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.509380:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-14, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.509417:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x25:0x0](ffff920a50d2b248) 00000080:00200000:2.0:1788572466.509486:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x25:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:2.0:1788572466.509491:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x25:0x0](ffff920a50d2b248) 00000080:00010000:2.0:1788572466.509495:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x25:0x0](ffff920a50d2b248) for lock 0x8364c713e25bb57a 00000080:00010000:2.0:1788572466.509504:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x25:0x0](ffff920a50d2b248) 00000080:00200008:2.0:1788572466.509520:0:164272:0:(file.c:1090:ll_file_open()) START file f-14:[0x240002729:0x25:0x0](ffff920a50d2b248), flags 100102 00000002:00100000:2.0:1788572466.509542:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4e00 x1875444569552384/t25769803807(25769803807) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:2.0:1788572466.509578:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb57a from it ffff920a479e9240 00010000:00010000:2.0:1788572466.509583:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.509654:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb57a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x25:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf10e expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.509675:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x25:0x0](ffff920a50d2b248), inode mode 81a4 mask 40 00000080:00200008:2.0:1788572466.509705:0:164272:0:(file.c:479:ll_file_release()) START file f-14:[0x240002729:0x25:0x0](ffff920a50d2b248), flags 100002 00000002:00100000:2.0:1788572466.509746:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac4e00 x1875444569552384/t25769803807(25769803807) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:2.0:1788572466.509757:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:2.0:1788572466.509762:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac7480 x1875444569552640/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:2.0:1788572466.509781:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552640:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.509892:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.512391:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552640:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.512427:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac4e00 x1875444569552384/t25769803807(25769803807) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1788572466.512506:0:164272:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572466.512524:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572466.512543:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788572466.512564:0:164272:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:2.0:1788572466.512571:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572466.512577:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572466.512581:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:2.0:1788572466.512619:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7300dd3bf8 00010000:00010000:2.0:1788572466.512624:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:2.0:1788572466.512634:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:2.0:1788572466.512648:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.512660:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=25 00000080:00200000:2.0:1788572466.512667:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.512671:0:164272:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:2.0:1788572466.512675:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.512689:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.512693:0:164272:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-15, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), file ffff920a53456600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:2.0:1788572466.512702:0:164272:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-15, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:2.0:1788572466.512722:0:164272:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-15,[0x240002729:0x26:0x0]) in obj [0x240002729:0x16:0x0], intent: open|creat flags 0100103 00010000:00010000:2.0:1788572466.512772:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.512780:0:164272:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572466.512792:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572466.512810:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5500 x1875444569552896/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572466.512820:0:164272:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164272 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572466.512839:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552896:192.168.202.157@tcp:101:createmany.0 00000100:00100000:2.0:1788572466.512972:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dcc0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.517141:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569552896:192.168.202.157@tcp:101:createmany.0 00010000:00010000:2.0:1788572466.517171:0:164272:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x16:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517202:0:164272:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x26:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517211:0:164272:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517223:0:164272:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517234:0:164272:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517247:0:164272:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517262:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517270:0:164272:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572466.517417:0:164272:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a60ac5500 x1875444569552896/t25769803809(25769803809) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572466.517432:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac5500 x1875444569552896/t25769803809(25769803809) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572466.517448:0:164272:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1788572466.517461:0:164272:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.517473:0:164272:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/38 (0 0) 00000080:00200000:2.0:1788572466.517479:0:164272:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x16:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1788572466.517493:0:164272:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-15, dir=[0x240002729:0x16:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:2.0:1788572466.517525:0:164272:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x26:0x0](ffff920a50d2d348) 00000080:00200000:3.0:1788572466.518872:0:164272:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x26:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572466.518900:0:164272:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x26:0x0](ffff920a50d2d348) 00000080:00010000:3.0:1788572466.518905:0:164272:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x26:0x0](ffff920a50d2d348) for lock 0x8364c713e25bb581 00000080:00010000:3.0:1788572466.518918:0:164272:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x26:0x0](ffff920a50d2d348) 00000080:00200008:3.0:1788572466.518934:0:164272:0:(file.c:1090:ll_file_open()) START file f-15:[0x240002729:0x26:0x0](ffff920a50d2d348), flags 100102 00000002:00100000:3.0:1788572466.518965:0:164272:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac5500 x1875444569552896/t25769803809(25769803809) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572466.519004:0:164272:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb581 from it ffff920a479e9240 00010000:00010000:3.0:1788572466.519010:0:164272:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572466.519022:0:164272:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.519045:0:164272:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x26:0x0](ffff920a50d2d348), inode mode 81a4 mask 40 00000080:00200008:3.0:1788572466.519078:0:164272:0:(file.c:479:ll_file_release()) START file f-15:[0x240002729:0x26:0x0](ffff920a50d2d348), flags 100002 00000002:00100000:3.0:1788572466.519126:0:164272:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac5500 x1875444569552896/t25769803809(25769803809) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572466.519137:0:164272:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572466.519142:0:164272:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3d180 x1875444569553152/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572466.519162:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569553152:192.168.202.157@tcp:35:createmany.0 00000100:00100000:3.0:1788572466.519242:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865e40 going to sleep for 11 seconds 00010000:00010000:1.0:1788572466.587924:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572466.591770:0:164272:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a478bf100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569555200:192.168.202.157@tcp:35:createmany.0 00000100:00100000:2.0:1788572466.591878:0:164272:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572466.597177:0:164272:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a478bf100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164272:1875444569555200:192.168.202.157@tcp:35:createmany.0 00000002:00100000:2.0:1788572466.597215:0:164272:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a478bf480 x1875444569554944/t25769803817(25769803817) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572466.642814:0:164277:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.642846:0:164277:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.642866:0:164277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000100:00100000:1.0:1788572466.665280:0:164277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34e00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164277:1875444569555712:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572466.665306:0:164277:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: --/CR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665330:0:164277:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: --/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665338:0:164277:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: --/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665349:0:164277:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665364:0:164277:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665378:0:164277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665387:0:164277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665395:0:164277:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572466.665406:0:164277:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a45b34e00 x1875444569555712/t25769803820(25769803820) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572466.665419:0:164277:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34e00 x1875444569555712/t25769803820(25769803820) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572466.665431:0:164277:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572466.665444:0:164277:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665457:0:164277:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb51f lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bef47 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572466.665469:0:164277:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb51f lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bef47 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572466.665478:0:164277:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665484:0:164277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665491:0:164277:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.665558:0:164277:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:1.0:1788572466.665566:0:164277:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x240002729:0x16:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572466.665641:0:164277:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x18:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1788572466.665647:0:164277:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x18:0x0](ffff920a50d463c8) for lock 0x8364c713e25bb51f 00000080:00010000:1.0:1788572466.665654:0:164277:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb51f from it ffffaf7300dd3bf0 00010000:00010000:1.0:1788572466.665658:0:164277:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb51f lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bef47 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572466.665668:0:164277:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb51f lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bef47 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1788572466.665695:0:164277:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34e00 x1875444569555712/t25769803820(25769803820) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572466.665790:0:164277:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x240002729:0x18:0x0](ffff920a50d463c8) cmd=800466af arg=7fffa363b93c 00000080:00200008:1.0:1788572466.665828:0:164277:0:(file.c:479:ll_file_release()) START file f-1:[0x240002729:0x18:0x0](ffff920a50d463c8), flags 100000 00000002:00100000:1.0:1788572466.665869:0:164277:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b34e00 x1875444569555712/t25769803820(25769803820) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572466.665884:0:164277:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572466.665901:0:164277:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350a80 x1875444569555968/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572466.665923:0:164277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164277:1875444569555968:192.168.202.157@tcp:35:lfs.0 00000100:00100000:1.0:1788572466.665997:0:164277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6d80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572466.668470:0:160929:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788572466.668485:0:160929:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.668497:0:160929:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a51a70e00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:2.0:1788572466.668503:0:160929:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.668512:0:160929:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.668522:0:160929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.668550:0:160929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb51f lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bef47 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000100:00100000:1.0:1788572466.668702:0:164277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164277:1875444569555968:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572466.668735:0:164277:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b34e00 x1875444569555712/t25769803820(25769803820) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:1.0:1788572466.668917:0:164277:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x16:0x0](ffff920a50d44b08), flags 304000 00000002:00100000:1.0:1788572466.668964:0:164277:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b35180 x1875444569555456/t25769803819(25769803819) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572466.668975:0:164277:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572466.668980:0:164277:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350a80 x1875444569556224/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572466.669002:0:164277:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164277:1875444569556224:192.168.202.157@tcp:35:lfs.0 00000100:00100000:1.0:1788572466.669077:0:164277:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6d80 going to sleep for 11 seconds 00010000:00010000:2.0:1788572466.669216:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 2/0,0 mode: --/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.669224:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572466.669255:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 1/0,0 mode: --/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.669263:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb5ab lrc: 0/0,0 mode: --/PR res: [0x240002729:0x18:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf1e0 expref: -99 pid: 164277 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572466.669277:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572466.669280:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572466.669282:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572466.669291:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572466.669303:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569556352:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572466.673676:0:164277:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164277:1875444569556224:192.168.202.157@tcp:35:lfs.0 00000002:00100000:1.0:1788572466.673714:0:164277:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b35180 x1875444569555456/t25769803819(25769803819) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572482 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:3.0:1788572466.673892:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569556352:192.168.202.157@tcp:103:ldlm_bl.0 00000080:00200000:1.0:1788572466.684435:0:164279:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572466.684468:0:164279:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.684487:0:164279:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788572466.684504:0:164279:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:1.0:1788572466.684511:0:164279:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572466.684519:0:164279:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572466.684522:0:164279:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb503 00000080:00010000:1.0:1788572466.684530:0:164279:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb503 from it ffffaf7301657b58 00010000:00010000:1.0:1788572466.684534:0:164279:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:1.0:1788572466.684542:0:164279:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816a00/0x8364c713e25bb503 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beec2 expref: -99 pid: 163782 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:1.0:1788572466.684558:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572466.684567:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80b.replay-single, flags=8 00000080:00200008:1.0:1788572466.684576:0:164279:0:(file.c:6296:ll_getattr_dentry()) START file d80b.replay-single:[0x200027501:0x4:0x0](ffff920a50d442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572466.684580:0:164279:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x4:0x0](ffff920a50d442c8),name=d80b.replay-single 00000002:00010000:1.0:1788572466.684619:0:164279:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x4:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572466.684627:0:164279:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/4 (0 0) 00010000:00010000:1.0:1788572466.684667:0:164279:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5b2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164279 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.684674:0:164279:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5b2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164279 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.684683:0:164279:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5b2 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164279 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572466.684714:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569556480:192.168.202.157@tcp:101:rm.0 00000100:00100000:3.0:1788572466.810344:0:164279:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559168:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572466.810421:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x1a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572466.810436:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572466.810465:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x1a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572466.810474:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x1a:0x0](ffff920a50d400c8) 00000080:00200000:3.0:1788572466.810554:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.810565:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-16, flags=0 00000080:00200000:3.0:1788572466.810571:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.810576:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-16, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:3.0:1788572466.810656:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb588 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf154 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.810677:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x27:0x0]: losing layout lock 00010000:00010000:3.0:1788572466.810720:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb588 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf154 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572466.810728:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572466.810731:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb588 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x27:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf154 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572466.810755:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572466.810763:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3c700 x1875444569559424/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572466.810790:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559424:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572466.810873:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.816149:0:164279:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559424:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572466.816212:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x27:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572466.816226:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572466.816254:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x27:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572466.816262:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x27:0x0](ffff920a50d421c8) 00000080:00200000:3.0:1788572466.816347:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.816359:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-4, flags=0 00000080:00200000:3.0:1788572466.816365:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.816370:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-4, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:3.0:1788572466.816407:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.816428:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x1b:0x0]: losing layout lock 00010000:00010000:3.0:1788572466.816471:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572466.816478:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572466.816482:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb534 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109befb0 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572466.816505:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572466.816513:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3c000 x1875444569559680/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572466.816533:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559680:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572466.816636:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.820581:0:164279:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559680:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572466.820696:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572466.820711:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572466.820737:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572466.820745:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x1b:0x0](ffff920a50d4b248) 00000080:00200000:3.0:1788572466.820839:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.820851:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-15, flags=0 00000080:00200000:3.0:1788572466.820857:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.820862:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-15, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:3.0:1788572466.820922:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.820943:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x26:0x0]: losing layout lock 00010000:00010000:3.0:1788572466.820985:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572466.821007:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572466.821010:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816c00/0x8364c713e25bb581 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x26:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf131 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572466.821033:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572466.821041:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ca80 x1875444569559936/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572466.821063:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559936:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572466.821132:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.824662:0:164279:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ca80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569559936:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572466.824723:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x26:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572466.824737:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572466.824764:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x26:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572466.824771:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x26:0x0](ffff920a50d2d348) 00000080:00200000:3.0:1788572466.824865:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.824886:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-0, flags=0 00000080:00200000:3.0:1788572466.824903:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.824908:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-0, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:3.0:1788572466.824952:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bef24 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.824970:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x17:0x0]: losing layout lock 00010000:00010000:3.0:1788572466.825315:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bef24 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572466.825324:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572466.825326:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec7600/0x8364c713e25bb518 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x17:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bef24 expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572466.825353:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572466.825363:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ed80 x1875444569560192/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572466.825385:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569560192:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572466.825466:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865900 going to sleep for 11 seconds 00000100:00100000:3.0:1788572466.836517:0:164279:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3ed80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569560192:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572466.836580:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x17:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:3.0:1788572466.836717:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:3.0:1788572466.836749:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x17:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572466.836756:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x17:0x0](ffff920a50d45b88) 00000080:00200000:3.0:1788572466.836848:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572466.836858:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-8, flags=0 00000080:00200000:3.0:1788572466.836864:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572466.836868:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-8, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:3.0:1788572466.836918:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb550 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf03c expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572466.836937:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x1f:0x0]: losing layout lock 00010000:00010000:3.0:1788572466.836978:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb550 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf03c expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572466.836985:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572466.837822:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb550 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x1f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf03c expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:3.0:1788572466.837859:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572466.837871:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51b3ce00 x1875444569560448/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572466.838860:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3ce00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569560448:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572466.838970:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d600 going to sleep for 11 seconds 00000080:00200000:2.0:1788572466.849329:0:164279:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x1f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572466.849343:0:164279:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572466.849412:0:164279:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x1f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572466.849420:0:164279:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x1f:0x0](ffff920a50d45348) 00000080:00200000:2.0:1788572466.849504:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572466.849517:0:164279:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-9, flags=0 00000080:00200000:2.0:1788572466.849523:0:164279:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x16:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572466.849528:0:164279:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-9, dir=[0x240002729:0x16:0x0](ffff920a50d44b08) 00010000:00010000:2.0:1788572466.849567:0:164279:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb557 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf05f expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572466.849616:0:164279:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x20:0x0]: losing layout lock 00010000:00010000:2.0:1788572466.849663:0:164279:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb557 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf05f expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572466.849671:0:164279:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572466.849675:0:164279:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb557 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x20:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf05f expref: -99 pid: 164272 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572466.849708:0:164279:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572466.849717:0:164279:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6680 x1875444569560704/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572466.849745:0:164279:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164279:1875444569560704:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572466.849832:0:164279:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d900 going to sleep for 11 seconds 00010000:00010000:1.0:1788572466.968913:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572466.968923:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.968933:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a51a71400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572466.968938:0:160930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.968947:0:160930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.968956:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.968978:0:160930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb50a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109beeec expref: -99 pid: 164265 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572466.969032:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 2/0,0 mode: --/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.969040:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572466.969068:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.969076:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb5d5 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x16:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf487 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572466.969094:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572466.969096:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572466.969098:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572466.969108:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572466.969126:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569563392:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572466.984320:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569563392:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572466.987800:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572466.987969:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584432896 00000100:00100000:2.0:1788572466.987979:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584432896 00000100:00100000:2.0:1788572466.988035:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 7 00000100:00100000:2.0:1788572466.988044:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:189111:x1875444584432896:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572466.988061:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5b2 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x4:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bf1f5 expref: -99 pid: 164279 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572466.988130:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572466.988136:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a51b3d500 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:189111:x1875444584432896:12345-192.168.202.157@tcp:104: Request processed in 93us (345us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572466.988143:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 7 00000100:00100000:1.0:1788572466.988819:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569564160:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:3.0:1788572468.188486:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569564800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572468.188575:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572468.188938:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565312:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572468.188970:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:3.0:1788572468.188976:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569564928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572468.189235:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572468.195440:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3d880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569564928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572468.195466:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3c380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572468.195476:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569564800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572468.195488:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572468.195498:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569565312:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572470.684562:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572473.309743:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569565440:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.309832:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565696:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.309871:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565952:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.309901:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572473.309907:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565568:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.309931:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565824:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.323963:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569565440:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.335678:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565568:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.335696:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565824:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.335706:0:2400:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 25769803845 gen 1 00000100:00100000:3.0:1788572473.336128:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565696:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572473.336145:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569565952:192.168.202.157@tcp:400:kworker.0 00000080:00200000:2.0:1788572474.292300:0:164859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572474.292334:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572474.292400:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572474.292414:0:164859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572474.292425:0:164859:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572474.292452:0:164859:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566080:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572474.292541:0:164859:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572474.303977:0:164859:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566080:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572474.304007:0:164859:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304021:0:164859:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304033:0:164859:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304115:0:164859:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304130:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304138:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304146:0:164859:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572474.304158:0:164859:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac5880 x1875444569566080/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572490 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572474.304172:0:164859:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304185:0:164859:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572474.304191:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572474.304220:0:164859:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:1.0:1788572474.304226:0:164859:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5e3 00000080:00010000:1.0:1788572474.304231:0:164859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572474.304235:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2bb58 00010000:00010000:1.0:1788572474.304239:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304248:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572474.304275:0:164859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572474.304295:0:164859:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=13 00000080:00200000:1.0:1788572474.304299:0:164859:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:1.0:1788572474.304319:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572474.304358:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304366:0:164859:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.304376:0:164859:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572474.304396:0:164859:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566208:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572474.304476:0:164859:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572474.310712:0:164859:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566208:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572474.310740:0:164859:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.310754:0:164859:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.310760:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.310766:0:164859:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.310775:0:164859:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.311927:0:164859:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572474.311935:0:164859:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.311943:0:164859:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.311953:0:164859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.311961:0:164859:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572474.311968:0:164859:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572474.311976:0:164859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78800/0x8364c713e25bb5ea lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164859 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572474.312003:0:164859:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a48350000 x1875444569566208/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572490 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572474.312018:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572474.312124:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572474.312140:0:164859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:1.0:1788572474.312150:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2b930 00010000:00010000:1.0:1788572474.312154:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572474.312163:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572474.312255:0:164859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572474.312271:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572474.312281:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572474.312290:0:164859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572474.312297:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572474.312301:0:164859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572474.312304:0:164859:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5e3 00000080:00010000:1.0:1788572474.312309:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2bbe8 00010000:00010000:1.0:1788572474.312312:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572474.312319:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572474.312327:0:164859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572474.312334:0:164859:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=16392 00000080:00200008:1.0:1788572474.312358:0:164859:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572474.312363:0:164859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572474.312370:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572474.312375:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572474.312383:0:164859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572474.312390:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572474.312395:0:164859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572474.312397:0:164859:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5e3 00000080:00010000:1.0:1788572474.312401:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2bcb8 00010000:00010000:1.0:1788572474.312404:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572474.312411:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788572474.313330:0:164859:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572474.313338:0:164859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572474.313351:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572474.313362:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572474.313374:0:164859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572474.313381:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572474.313387:0:164859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572474.313390:0:164859:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5e3 00000080:00010000:1.0:1788572474.313395:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2bcb8 00010000:00010000:1.0:1788572474.313398:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572474.313407:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572474.313543:0:164859:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572474.313552:0:164859:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572474.313558:0:164859:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572474.313566:0:164859:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572474.313572:0:164859:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572474.313575:0:164859:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572474.313577:0:164859:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5e3 00000080:00010000:1.0:1788572474.313580:0:164859:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5e3 from it ffffaf7300e2bbb0 00010000:00010000:1.0:1788572474.313583:0:164859:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572474.313619:0:164859:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572474.313628:0:164859:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572474.313636:0:164859:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00000080:00200080:1.0:1788572474.313652:0:164859:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c03066f0 arg=7ffdb7e30960 00000080:00200000:1.0:1788572474.313665:0:164859:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) name=d80c.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:1.0:1788572474.313691:0:164859:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572474.313742:0:164859:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:1.0:1788572474.313751:0:164859:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572474.313753:0:164859:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817a00/0x8364c713e25bb5e3 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf575 expref: -99 pid: 164859 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:1.0:1788572474.313791:0:164859:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572474.313804:0:164859:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350000 x1875444569566464/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572474.313830:0:164859:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566464:192.168.202.157@tcp:36:lfs.0 00000100:00100000:1.0:1788572474.313932:0:164859:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572474.331674:0:164859:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164859:1875444569566464:192.168.202.157@tcp:36:lfs.0 00000080:00200000:1.0:1788572474.331723:0:164859:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00200000:1.0:1788572474.331730:0:164859:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00200008:1.0:1788572474.331764:0:164859:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00000100:00100000:3.0:1788572475.333881:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36a00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566592:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.333910:0:164868:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333923:0:164868:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333935:0:164868:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333949:0:164868:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333962:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333971:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.333979:0:164868:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.333991:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a45b36a00 x1875444569566592/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572475.334005:0:164868:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.334019:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572475.334025:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572475.334051:0:164868:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572475.334057:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572475.334062:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572475.334065:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e2bb58 00010000:00010000:3.0:1788572475.334069:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.334083:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572475.334109:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572475.334125:0:164868:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=29 00000080:00200000:3.0:1788572475.334129:0:164868:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:3.0:1788572475.334150:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:3.0:1788572475.334187:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.334195:0:164868:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.334204:0:164868:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572475.334225:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566720:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572475.349948:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566720:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.349975:0:164868:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.349998:0:164868:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350007:0:164868:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350023:0:164868:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350034:0:164868:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350062:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350070:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350077:0:164868:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.350088:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a529b1180 x1875444569566720/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572475.350100:0:164868:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350111:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/7 (0 0) 00000080:00200000:3.0:1788572475.350118:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572475.350146:0:164868:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00200000:3.0:1788572475.350150:0:164868:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x7:0x0](ffff920a50d49988): rc = 0 00000080:00010000:3.0:1788572475.350154:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb5f8 00000080:00010000:3.0:1788572475.350167:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:3.0:1788572475.350171:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f8 from it ffffaf7300e2ba28 00010000:00010000:3.0:1788572475.350175:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350183:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572475.350214:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572475.350231:0:164868:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), flags=13 00000080:00200000:3.0:1788572475.350235:0:164868:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), intent=getattr 00000002:00010000:3.0:1788572475.350256:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572475.350295:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350302:0:164868:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.350323:0:164868:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572475.350344:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566848:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572475.350560:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572475.361909:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566848:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.361946:0:164868:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.361961:0:164868:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.361972:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.361980:0:164868:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.361989:0:164868:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.361995:0:164868:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec7000 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572475.362000:0:164868:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362008:0:164868:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362015:0:164868:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 5/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362022:0:164868:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572475.362030:0:164868:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362036:0:164868:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7000/0x8364c713e25bb5ff lrc: 0/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.362054:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a529b1180 x1875444569566848/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572475.362068:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572475.362095:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/7 (0 0) 00000080:00200000:3.0:1788572475.362169:0:164868:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572475.362184:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572475.362207:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.362219:0:164868:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572475.362226:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572475.362232:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572475.362235:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572475.362240:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e2bbe8 00010000:00010000:3.0:1788572475.362244:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.362253:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572475.362268:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572475.362276:0:164868:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=16408 00000080:00200000:3.0:1788572475.362281:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572475.362294:0:164868:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), flags=16392 00000080:00200000:3.0:1788572475.362298:0:164868:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), intent=getattr 00000002:00010000:3.0:1788572475.362310:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572475.362349:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362356:0:164868:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.362366:0:164868:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572475.362386:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566976:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572475.362477:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572475.365049:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569566976:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.365073:0:164868:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365089:0:164868:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365096:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365103:0:164868:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365112:0:164868:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365118:0:164868:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec6600 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572475.365122:0:164868:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365128:0:164868:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365135:0:164868:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 5/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365142:0:164868:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572475.365147:0:164868:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365153:0:164868:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6600/0x8364c713e25bb606 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.365169:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a529b1180 x1875444569566976/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572475.365182:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572475.365205:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/7 (0 0) 00000080:00200000:3.0:1788572475.365268:0:164868:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572475.365282:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572475.365291:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.365301:0:164868:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572475.365309:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572475.365313:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572475.365316:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572475.365322:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e2bb58 00010000:00010000:3.0:1788572475.365325:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572475.365334:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572475.365348:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572475.365355:0:164868:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=13 00000080:00200008:3.0:1788572475.365365:0:164868:0:(file.c:6296:ll_getattr_dentry()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572475.365369:0:164868:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988),name=d80c.replay-single 00000002:00010000:3.0:1788572475.365380:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572475.365385:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/7 (0 0) 00010000:00010000:3.0:1788572475.365418:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365424:0:164868:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.365432:0:164868:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572475.365451:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569567104:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572475.365523:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990180 going to sleep for 11 seconds 00000100:00100000:3.0:1788572475.373641:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569567104:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.373669:0:164868:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373687:0:164868:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373695:0:164868:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373713:0:164868:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373725:0:164868:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373737:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373746:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373754:0:164868:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.373765:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a529b1180 x1875444569567104/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572491 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572475.373779:0:164868:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.373788:0:164868:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/7 (0 0) 00000080:00200000:3.0:1788572475.373795:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572475.373898:0:164868:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00010000:2.0:1788572475.373907:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb60d 00000080:00010000:2.0:1788572475.373915:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:2.0:1788572475.373918:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb60d from it ffffaf7300e2bcb8 00010000:00010000:2.0:1788572475.373922:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.373931:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1788572475.374573:0:164868:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572475.374581:0:164868:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572475.374625:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572475.374638:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572475.374649:0:164868:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572475.374663:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572475.374669:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572475.374672:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:2.0:1788572475.374677:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e2bcb8 00010000:00010000:2.0:1788572475.374680:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572475.374689:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572475.374842:0:164868:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572475.374861:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572475.374869:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572475.374878:0:164868:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572475.374884:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572475.374889:0:164868:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572475.374892:0:164868:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:2.0:1788572475.374896:0:164868:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e2bbf8 00010000:00010000:2.0:1788572475.374899:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572475.374905:0:164868:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572475.374915:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572475.374922:0:164868:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=267 00000080:00200000:2.0:1788572475.374928:0:164868:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 44 00000080:00200008:2.0:1788572475.374935:0:164868:0:(file.c:1090:ll_file_open()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), flags 300000 00000002:00010000:2.0:1788572475.374950:0:164868:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:2.0:1788572475.374997:0:164868:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572475.375005:0:164868:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572475.375016:0:164868:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572475.375027:0:164868:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac4380 x1875444569567360/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572475.375037:0:164868:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572475.375058:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569567360:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572475.375233:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036600 going to sleep for 11 seconds 00000100:00100000:3.0:1788572475.383516:0:164868:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569567360:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572475.383541:0:164868:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383559:0:164868:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383566:0:164868:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383574:0:164868:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383583:0:164868:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383611:0:164868:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572475.383615:0:164868:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383623:0:164868:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383631:0:164868:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383638:0:164868:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572475.383645:0:164868:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572475.383652:0:164868:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb614 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 164868 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572475.383671:0:164868:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a60ac4380 x1875444569567360/t347892351008(347892351008) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572491 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572475.383690:0:164868:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4380 x1875444569567360/t347892351008(347892351008) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572491 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572475.383708:0:164868:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572475.383905:0:164868:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000002:00100000:2.0:1788572475.383932:0:164868:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac4380 x1875444569567360/t347892351008(347892351008) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572491 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:2.0:1788572475.384029:0:164868:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) cmd=c03066f0 arg=7ffd021eef80 00000080:00200000:2.0:1788572475.384045:0:164868:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00000020:00100000:2.0:1788572475.384105:0:164868:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572475.384112:0:164868:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5c00 x1875444569567616/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572475.384135:0:164868:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164868:1875444569567616:192.168.202.157@tcp:36:lfs.0 00000100:00100000:2.0:1788572475.384221:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572475.390879:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572475.390961:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584442880 00000100:00100000:3.0:1788572475.390972:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584442880 00000100:00100000:3.0:1788572475.391033:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 8 00000100:00100000:3.0:1788572475.391056:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a60ac6d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584442880:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572475.391075:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572475.391158:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572475.391165:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a60ac6d80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584442880:12345-192.168.202.157@tcp:104: Request processed in 113us (296us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572475.391173:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 8 00010000:00010000:1.0:1788572475.391429:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572475.391437:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391446:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec6400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572475.391452:0:160930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391460:0:160930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391469:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 3/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391500:0:160930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572475.391547:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 2/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391554:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572475.391606:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 1/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391616:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6400/0x8364c713e25bb60d lrc: 0/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bf601 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572475.391631:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572475.391632:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572475.391634:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572475.391649:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569567744:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572475.404661:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569567744:192.168.202.157@tcp:103:ldlm_bl.0 00000080:00200008:2.0:1788572476.701458:0:164877:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572476.701480:0:164877:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572476.701515:0:164877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.701541:0:164877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572476.701559:0:164877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1788572476.701574:0:164877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572476.703268:0:164877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572476.703285:0:164877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:2.0:1788572476.703302:0:164877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf730168fcb8 00010000:00010000:2.0:1788572476.703310:0:164877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1788572476.703326:0:164877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200008:2.0:1788572476.708204:0:164877:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:2.0:1788572476.708220:0:164877:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572476.708257:0:164877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.708285:0:164877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572476.708304:0:164877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1788572476.708313:0:164877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572476.708323:0:164877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572476.708327:0:164877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:2.0:1788572476.708336:0:164877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf730168fcb8 00010000:00010000:2.0:1788572476.708341:0:164877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1788572476.708352:0:164877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788572476.708531:0:164877:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572476.708539:0:164877:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.708545:0:164877:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788572476.708553:0:164877:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1788572476.708559:0:164877:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572476.708562:0:164877:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572476.708565:0:164877:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:2.0:1788572476.708568:0:164877:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf730168fbb0 00010000:00010000:2.0:1788572476.708570:0:164877:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1788572476.708579:0:164877:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1788572476.708998:0:164877:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572476.709011:0:164877:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:2.0:1788572476.713011:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:2.0:1788572476.713362:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00800000:00000004:2.0:1788572476.713376:0:164877:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9224, max_age 9234 00000100:00100000:2.0:1788572476.713440:0:164877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569567872:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572476.713675:0:164877:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572476.725015:0:164877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569567872:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572476.725173:0:164877:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317221/353529 objects 1023634/1024000 00000080:00200080:2.0:1788572476.725444:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:2.0:1788572476.725469:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00800000:00000004:2.0:1788572476.725478:0:164877:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9224, max_age 9234 00000100:00100000:2.0:1788572476.725531:0:164877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568000:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572476.725679:0:164877:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:2.0:1788572476.737677:0:164877:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0001-mdc-ffff920a47b93800: committing for last_committed 25769803846 gen 1 00000100:00100000:2.0:1788572476.737707:0:164877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568000:192.168.202.157@tcp:41:lfs.0 00800000:00000004:2.0:1788572476.737754:0:164877:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317318/353529 objects 1023722/1024000 00000080:00200080:3.0:1788572476.757533:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757572:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757676:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757683:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757688:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757691:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757694:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757697:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757701:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757704:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757707:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757711:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757715:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757718:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757722:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757725:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757729:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757807:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757822:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757826:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757829:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757833:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757837:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757840:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757844:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757847:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757851:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757983:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.757992:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.757996:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758000:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758004:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758008:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758012:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758016:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758019:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758024:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758028:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758032:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758035:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758040:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758171:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758179:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758183:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758187:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758191:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758194:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758198:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758202:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758205:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758209:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758213:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758217:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758220:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758224:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758228:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758232:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758237:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758240:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758244:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758248:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758251:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758255:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758259:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758263:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758266:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758269:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758273:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758277:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758280:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758284:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758287:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758290:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758293:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758296:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758300:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758304:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758307:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758310:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758314:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758318:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758321:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758325:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758328:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758332:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758335:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758340:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758343:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758347:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758350:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758354:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758357:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758361:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758364:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758368:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758371:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758375:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758378:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758382:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758385:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758389:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758392:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758395:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758398:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758402:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758405:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.758409:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.758412:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762621:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762651:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762657:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762663:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762667:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762672:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762676:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762680:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762685:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762688:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762693:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762696:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762699:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762702:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762706:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762709:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762713:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00800000:00000080:3.0:1788572476.762716:0:164877:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00000080:00200080:3.0:1788572476.762720:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:3.0:1788572476.762727:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43164800 uarg=0000000000000000 00020000:00000004:3.0:1788572476.766019:0:164877:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9224, max_age 9234 00000100:00100000:3.0:1788572476.766224:0:164877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568128:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572476.766426:0:164877:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990a80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572476.782185:0:164877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568128:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572476.782227:0:164877:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261721/262144 00000080:00200080:1.0:1788572476.782511:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.782534:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00020000:00000004:1.0:1788572476.782543:0:164877:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9224, max_age 9234 00000100:00100000:1.0:1788572476.782618:0:164877:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568256:192.168.202.157@tcp:13:lfs.0 00000100:00100000:1.0:1788572476.782744:0:164877:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572476.807327:0:164877:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164877:1875444569568256:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572476.807368:0:164877:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261737/262144 00000080:00200080:1.0:1788572476.807462:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807484:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807491:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807494:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807498:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807502:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807505:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807508:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807511:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807514:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807518:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807521:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807524:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807527:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807530:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807533:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807536:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807539:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807542:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807545:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807548:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807552:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807555:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807559:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807562:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807565:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807568:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807571:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807574:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807577:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807580:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807583:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807616:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807621:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807625:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807627:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807631:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807634:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807637:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807640:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807643:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807646:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807649:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807652:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807655:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807658:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807662:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807665:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807669:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807672:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807675:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807678:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807681:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807684:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807687:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807690:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807693:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807696:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807699:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807702:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807705:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807708:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807712:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807714:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807717:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807720:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807724:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807726:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807730:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807846:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807865:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807870:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807874:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807878:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.807882:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.807885:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808017:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808024:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808028:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808031:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808035:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808038:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808139:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808146:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808152:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808155:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808159:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808162:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808166:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808169:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808172:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808175:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808179:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808182:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808185:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808188:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808192:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808195:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808199:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808202:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808205:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808208:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808212:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808215:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808218:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808221:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808224:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808227:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808231:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808234:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808237:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808240:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808243:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808246:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808250:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808253:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808256:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808259:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808262:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808265:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808269:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808272:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808275:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808278:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572476.808282:0:164877:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffd24b53750 00020000:00000080:1.0:1788572476.808284:0:164877:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200008:1.0:1788572476.808301:0:164877:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:1.0:1788572476.829463:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:3.0:1788572476.913420:0:164884:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572476.913464:0:164884:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572476.913492:0:164884:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572476.913511:0:164884:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00000080:00200000:3.0:1788572476.913520:0:164884:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572476.913529:0:164884:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572476.913533:0:164884:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572476.913541:0:164884:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e33bd0 00010000:00010000:3.0:1788572476.913546:0:164884:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 8 lru_type: 0 00010000:00010000:3.0:1788572476.913556:0:164884:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572476.913576:0:164884:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572476.916672:0:164884:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=1536 00000080:00200000:3.0:1788572476.916692:0:164884:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572476.916722:0:164884:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572476.916791:0:164884:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572476.918745:0:164884:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00000080:00200000:3.0:1788572476.918771:0:164884:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572476.918787:0:164884:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572476.918792:0:164884:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572476.918804:0:164884:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e33cd0 00010000:00010000:3.0:1788572476.919289:0:164884:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 9 lru_type: 1 00010000:00010000:3.0:1788572476.919312:0:164884:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572476.919339:0:164884:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572476.919363:0:164884:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572476.919396:0:164884:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572476.919417:0:164884:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572476.919430:0:164884:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00000080:00200000:3.0:1788572476.919437:0:164884:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572476.919443:0:164884:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572476.919445:0:164884:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb5f1 00000080:00010000:3.0:1788572476.919449:0:164884:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb5f1 from it ffffaf7300e33db0 00010000:00010000:3.0:1788572476.919453:0:164884:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 10 lru_type: 1 00010000:00010000:3.0:1788572476.919461:0:164884:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000080:00200000:3.0:1788572476.919470:0:164884:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572476.919477:0:164884:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=33188, excl=1 00000080:00200000:3.0:1788572476.919482:0:164884:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) mode 100644 dev 0 00010000:00010000:3.0:1788572476.919519:0:164884:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572476.919625:0:164884:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:3.0:1788572476.919633:0:164884:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572476.919636:0:164884:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb5f1 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf5ad expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00000020:00100000:3.0:1788572476.919674:0:164884:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572476.919693:0:164884:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b0000 x1875444569568512/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572476.919724:0:164884:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164884:1875444569568512:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572476.921315:0:164884:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572476.928142:0:164884:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164884:1875444569568512:192.168.202.157@tcp:36:mcreate.0 00000080:00200000:3.0:1788572476.928220:0:164884:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0x8:0x0](ffff920a50d3d348) 00000080:00200000:3.0:1788572476.928277:0:164884:0:(namei.c:2070:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, unhashed 1 00000080:00200000:2.0:1788572476.963127:0:164887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572476.963169:0:164887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572476.963257:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.963271:0:164887:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572476.963284:0:164887:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572476.963318:0:164887:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569568640:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572476.963444:0:164887:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036480 going to sleep for 11 seconds 00000100:00100000:3.0:1788572476.987061:0:164887:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569568640:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572476.987099:0:164887:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987115:0:164887:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987129:0:164887:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987146:0:164887:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987162:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987171:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987179:0:164887:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572476.987193:0:164887:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac5880 x1875444569568640/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572492 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572476.987211:0:164887:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987231:0:164887:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572476.987240:0:164887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572476.987280:0:164887:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572476.987287:0:164887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb61b 00000080:00010000:3.0:1788572476.987295:0:164887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572476.987298:0:164887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb61b from it ffffaf7301507b58 00010000:00010000:3.0:1788572476.987301:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987316:0:164887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572476.987347:0:164887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572476.987375:0:164887:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:3.0:1788572476.987584:0:164887:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572476.987670:0:164887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572476.987730:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987738:0:164887:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572476.987748:0:164887:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164887 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572476.987776:0:164887:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569568768:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572476.995455:0:164887:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572477.002503:0:164887:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569568768:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572477.002541:0:164887:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002570:0:164887:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x8:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002579:0:164887:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002645:0:164887:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002658:0:164887:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002673:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002680:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002688:0:164887:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572477.002701:0:164887:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a529b3b80 x1875444569568768/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572492 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572477.002715:0:164887:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002726:0:164887:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/8 (0 0) 00000080:00200000:1.0:1788572477.002734:0:164887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572477.002760:0:164887:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x8:0x0], updating i_size 0 00000080:00200000:1.0:1788572477.002763:0:164887:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x8:0x0](ffff920a50d3d348): rc = 0 00000080:00200000:1.0:1788572477.002785:0:164887:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x8:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:1.0:1788572477.002792:0:164887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x8:0x0](ffff920a50d3d348) for lock 0x8364c713e25bb622 00000080:00010000:1.0:1788572477.002805:0:164887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x8:0x0](ffff920a50d3d348) 00000080:00010000:1.0:1788572477.002809:0:164887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb622 from it ffffaf7301507b20 00010000:00010000:1.0:1788572477.002812:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572477.002823:0:164887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572477.002882:0:164887:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x200027501:0x8:0x0](ffff920a50d3d348), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572477.002891:0:164887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x8:0x0](ffff920a50d3d348),name=fsa-oleg257-client.virtnet 00000002:00010000:1.0:1788572477.002921:0:164887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x8:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572477.002932:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572477.002945:0:164887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572477.002952:0:164887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572477.002958:0:164887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x8:0x0](ffff920a50d3d348) 00000080:00010000:1.0:1788572477.002962:0:164887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x8:0x0](ffff920a50d3d348) for lock 0x8364c713e25bb622 00000080:00010000:1.0:1788572477.002966:0:164887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb622 from it ffffaf7301507cb8 00010000:00010000:1.0:1788572477.002970:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572477.002978:0:164887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572477.003051:0:164887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572477.003060:0:164887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572477.003067:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572477.003075:0:164887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572477.003082:0:164887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572477.003085:0:164887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572477.003088:0:164887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb61b 00000080:00010000:1.0:1788572477.003092:0:164887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb61b from it ffffaf7301507c10 00010000:00010000:1.0:1788572477.003095:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572477.003104:0:164887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572477.003112:0:164887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572477.003123:0:164887:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:1.0:1788572477.003129:0:164887:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572477.003136:0:164887:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572477.003142:0:164887:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572477.003150:0:164887:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572477.003157:0:164887:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572477.003161:0:164887:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572477.003163:0:164887:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb61b 00000080:00010000:1.0:1788572477.003178:0:164887:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb61b from it ffffaf7301507d68 00010000:00010000:1.0:1788572477.003180:0:164887:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572477.003188:0:164887:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572477.003195:0:164887:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572477.003201:0:164887:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:1.0:1788572477.003276:0:164887:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572477.003305:0:164887:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572477.003315:0:164887:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x8:0x0]: losing layout lock 00010000:00010000:1.0:1788572477.003355:0:164887:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572477.003363:0:164887:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572477.003369:0:164887:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788572477.003372:0:164887:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb622 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x8:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf663 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572477.003391:0:164887:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816e00/0x8364c713e25bb61b lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf647 expref: -99 pid: 164887 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:1.0:1788572477.003413:0:164887:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572477.003422:0:164887:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352680 x1875444569569024/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572477.003451:0:164887:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569569024:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572477.003563:0:164887:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572477.018423:0:164887:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164887:1875444569569024:192.168.202.157@tcp:36:rm.0 00000080:00200000:1.0:1788572477.018513:0:164887:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:1.0:1788572477.018525:0:164887:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1788572477.018536:0:164887:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x8:0x0](ffff920a50d3d348) 00000100:00100000:3.0:1788572478.428314:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569280:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.428417:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569536:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.428444:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788572478.428449:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569569152:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.428474:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569408:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.428493:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.439047:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569569152:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.439191:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569408:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.439206:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.439220:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569280:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572478.439231:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569569536:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572478.876569:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572480.943614:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572481.949352:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572482.971818:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572483.548167:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569569920:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572483.548387:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569569792:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572483.548447:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569570176:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572483.562952:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569570048:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572483.563054:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569570304:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572483.564408:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569569920:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572483.564427:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569570176:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572483.564484:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569569792:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572483.567490:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569570048:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572483.573032:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569570304:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572484.004734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:3.0:1788572484.944227:0:164959:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 00010000:00010000:3.0:1788572485.019933:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572486.047479:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572486.056398:0:164968:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572486.056415:0:164968:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572486.056441:0:164968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572486.056505:0:164968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572486.056713:0:164968:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164968 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572486.056726:0:164968:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164968 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572486.058074:0:164968:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570432:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572486.058172:0:164968:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a42bc6540 going to sleep for 50 seconds 00000100:00100000:3.0:1788572486.060930:0:164968:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570432:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572486.060957:0:164968:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.060969:0:164968:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.060981:0:164968:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.060998:0:164968:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.061011:0:164968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.061018:0:164968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.061026:0:164968:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572486.061038:0:164968:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac7480 x1875444569570432/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572486.061053:0:164968:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.061066:0:164968:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572486.061073:0:164968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572486.061113:0:164968:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572486.061120:0:164968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.061125:0:164968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.061129:0:164968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf7301cdfcb8 00010000:00010000:3.0:1788572486.061132:0:164968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.061147:0:164968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572486.061901:0:164968:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572486.061919:0:164968:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.061936:0:164968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.061948:0:164968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.061962:0:164968:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572486.061970:0:164968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.061976:0:164968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.061979:0:164968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.061984:0:164968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf7301cdfcb8 00010000:00010000:3.0:1788572486.061987:0:164968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.061995:0:164968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.062104:0:164968:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.062113:0:164968:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.062120:0:164968:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.062134:0:164968:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.062141:0:164968:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.062145:0:164968:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.062147:0:164968:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.062150:0:164968:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf7301cdfbb0 00010000:00010000:3.0:1788572486.062153:0:164968:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.062161:0:164968:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572486.062171:0:164968:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572486.062178:0:164968:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:3.0:1788572486.062262:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:3.0:1788572486.062278:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00800000:00000004:3.0:1788572486.062285:0:164968:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9235, max_age 9243 00000100:00100000:3.0:1788572486.062321:0:164968:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570560:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572486.062393:0:164968:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990780 going to sleep for 50 seconds 00000100:00100000:3.0:1788572486.083272:0:164968:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570560:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572486.083307:0:164968:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317221/353529 objects 1023634/1024000 00000080:00200080:2.0:1788572486.083494:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.083515:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314f400 uarg=0000000000000000 00800000:00000004:2.0:1788572486.083523:0:164968:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9235, max_age 9243 00000100:00100000:2.0:1788572486.083561:0:164968:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570688:192.168.202.157@tcp:41:lfs.0 00000100:00100000:2.0:1788572486.083670:0:164968:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a42bc6540 going to sleep for 50 seconds 00000080:00200080:2.0:1788572486.098641:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098673:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098690:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098696:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098701:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098705:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098709:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098714:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098719:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098722:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098726:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098730:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098734:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098738:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098754:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098759:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098764:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098768:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098772:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098776:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098780:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098783:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098786:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098790:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098794:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098797:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098801:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098804:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098807:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098810:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098814:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098817:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098820:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098823:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098827:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098830:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098834:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098838:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098841:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098845:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098848:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098852:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098857:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098861:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098865:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098869:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098873:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098876:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098880:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098884:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098888:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098892:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098896:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098899:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098903:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098919:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098924:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098928:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098933:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098937:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098941:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098945:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098949:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098953:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098957:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098960:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098963:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098968:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098971:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098975:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098979:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098982:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098986:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098990:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.098993:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.098997:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099001:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099004:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099007:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099011:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099015:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099018:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099022:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099025:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099029:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099033:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099037:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099041:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099045:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099049:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099053:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099056:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099061:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099064:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099068:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099071:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099075:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099079:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099083:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099086:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099090:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099390:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099403:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099410:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099413:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099417:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099420:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099424:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099427:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099430:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099434:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099437:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099441:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099445:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099449:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099453:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099456:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099460:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099474:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099479:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099483:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099487:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099491:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00800000:00000080:2.0:1788572486.099495:0:164968:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00000080:00200080:2.0:1788572486.099499:0:164968:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff9f43fa60 00020000:00000080:2.0:1788572486.099505:0:164968:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4314ec00 uarg=0000000000000000 00020000:00000004:2.0:1788572486.099513:0:164968:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9235, max_age 9243 00000100:00100000:2.0:1788572486.099563:0:164968:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164968:1875444569570816:192.168.202.157@tcp:13:lfs.0 00000100:00100000:2.0:1788572486.099688:0:164968:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a42bc6540 going to sleep for 11 seconds 00000080:00200000:3.0:1788572486.185933:0:164975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.185980:0:164975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.186006:0:164975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572486.186029:0:164975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572486.186039:0:164975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.186049:0:164975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.186053:0:164975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.186063:0:164975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf730168fbd0 00010000:00010000:3.0:1788572486.186067:0:164975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572486.186079:0:164975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572486.186099:0:164975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572486.186116:0:164975:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=1536 00000080:00200000:3.0:1788572486.186120:0:164975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.186126:0:164975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.186132:0:164975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572486.186141:0:164975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572486.186147:0:164975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.186151:0:164975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.186153:0:164975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.186158:0:164975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf730168fcd0 00010000:00010000:3.0:1788572486.186161:0:164975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572486.186169:0:164975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572486.186177:0:164975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572486.186184:0:164975:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.186191:0:164975:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.186197:0:164975:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572486.186205:0:164975:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572486.186212:0:164975:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.186216:0:164975:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.186218:0:164975:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb629 00000080:00010000:3.0:1788572486.186222:0:164975:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb629 from it ffffaf730168fdb0 00010000:00010000:3.0:1788572486.186225:0:164975:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572486.186232:0:164975:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572486.186241:0:164975:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572486.186247:0:164975:0:(namei.c:2062:ll_create_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=33188, excl=1 00000080:00200000:3.0:1788572486.186251:0:164975:0:(namei.c:2017:ll_mknod()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) mode 100644 dev 0 00010000:00010000:3.0:1788572486.186283:0:164975:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572486.186346:0:164975:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572486.186355:0:164975:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572486.186357:0:164975:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817c00/0x8364c713e25bb629 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf68d expref: -99 pid: 164968 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:3.0:1788572486.186390:0:164975:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572486.186408:0:164975:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b1f80 x1875444569571200/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572486.186438:0:164975:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job mcreate:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164975:1875444569571200:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572486.186559:0:164975:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990780 going to sleep for 50 seconds 00000080:00200000:1.0:1788572486.218001:0:164978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572486.218115:0:164978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572486.218330:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.218346:0:164978:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.218362:0:164978:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572486.218400:0:164978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571328:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572486.218522:0:164978:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 50 seconds 00000100:00100000:1.0:1788572486.231300:0:164978:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571328:192.168.202.157@tcp:101:rm.0 00010000:00010000:1.0:1788572486.231348:0:164978:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231366:0:164978:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231382:0:164978:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231400:0:164978:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231415:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231424:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231432:0:164978:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572486.231446:0:164978:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48352300 x1875444569571328/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572486.231463:0:164978:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231489:0:164978:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572486.231498:0:164978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572486.231533:0:164978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:1.0:1788572486.231540:0:164978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb630 00000080:00010000:1.0:1788572486.231547:0:164978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572486.231550:0:164978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb630 from it ffffaf7301cefb58 00010000:00010000:1.0:1788572486.231554:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231569:0:164978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572486.231621:0:164978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572486.231645:0:164978:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=8 00000080:00200000:1.0:1788572486.231650:0:164978:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:1.0:1788572486.231676:0:164978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: fsa-oleg257-client.virtnet,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572486.231725:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231733:0:164978:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572486.231744:0:164978:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 164978 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572486.231770:0:164978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571456:192.168.202.157@tcp:101:rm.0 00000100:00100000:1.0:1788572486.231866:0:164978:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a76c0 going to sleep for 50 seconds 00000100:00100000:3.0:1788572486.235843:0:164978:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571456:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572486.235891:0:164978:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235929:0:164978:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x9:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235939:0:164978:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235952:0:164978:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235966:0:164978:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235981:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235990:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.235997:0:164978:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572486.236011:0:164978:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48352300 x1875444569571456/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 600/584 e 0 to 0 dl 1788572541 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572486.236025:0:164978:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.236036:0:164978:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/9 (0 0) 00000080:00200000:3.0:1788572486.236044:0:164978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572486.236073:0:164978:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x9:0x0], updating i_size 0 00000080:00200000:3.0:1788572486.236077:0:164978:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x9:0x0](ffff920a50d442c8): rc = 0 00000080:00200000:3.0:1788572486.236099:0:164978:0:(file.c:6843:ll_layout_conf()) [0x200027501:0x9:0x0]: layout version change: 4294967294 -> 4294967295 00000080:00010000:3.0:1788572486.236106:0:164978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x9:0x0](ffff920a50d442c8) for lock 0x8364c713e25bb637 00000080:00010000:3.0:1788572486.236120:0:164978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x9:0x0](ffff920a50d442c8) 00000080:00010000:3.0:1788572486.236125:0:164978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb637 from it ffffaf7301cefb20 00010000:00010000:3.0:1788572486.236128:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572486.236139:0:164978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572486.236188:0:164978:0:(file.c:6296:ll_getattr_dentry()) START file fsa-oleg257-client.virtnet:[0x200027501:0x9:0x0](ffff920a50d442c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:3.0:1788572486.236196:0:164978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x9:0x0](ffff920a50d442c8),name=fsa-oleg257-client.virtnet 00000002:00010000:3.0:1788572486.236225:0:164978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x9:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.236235:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.236247:0:164978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572486.236255:0:164978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.236261:0:164978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x9:0x0](ffff920a50d442c8) 00000080:00010000:3.0:1788572486.236264:0:164978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x9:0x0](ffff920a50d442c8) for lock 0x8364c713e25bb637 00000080:00010000:3.0:1788572486.236268:0:164978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb637 from it ffffaf7301cefcb8 00010000:00010000:3.0:1788572486.236272:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.236279:0:164978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.236352:0:164978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.236361:0:164978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.236368:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.236378:0:164978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572486.236385:0:164978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.236389:0:164978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.236392:0:164978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb630 00000080:00010000:3.0:1788572486.236395:0:164978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb630 from it ffffaf7301cefc10 00010000:00010000:3.0:1788572486.236398:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572486.236405:0:164978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.236414:0:164978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572486.236425:0:164978:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=fsa-oleg257-client.virtnet, flags=0 00000080:00200000:3.0:1788572486.236431:0:164978:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572486.236437:0:164978:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572486.236442:0:164978:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.236451:0:164978:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.236456:0:164978:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572486.236460:0:164978:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572486.236462:0:164978:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb630 00000080:00010000:3.0:1788572486.236466:0:164978:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb630 from it ffffaf7301cefd68 00010000:00010000:3.0:1788572486.236469:0:164978:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.236475:0:164978:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572486.236483:0:164978:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572486.236489:0:164978:0:(namei.c:2351:ll_unlink()) VFS Op:name=fsa-oleg257-client.virtnet, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:3.0:1788572486.236513:0:164978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572486.236540:0:164978:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572486.236550:0:164978:0:(vvp_object.c:117:vvp_conf_set()) [0x200027501:0x9:0x0]: losing layout lock 00010000:00010000:3.0:1788572486.236636:0:164978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.236645:0:164978:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572486.236651:0:164978:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:3.0:1788572486.236655:0:164978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78000/0x8364c713e25bb637 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bf6c5 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572486.236674:0:164978:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb630 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bf6a9 expref: -99 pid: 164978 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:3.0:1788572486.236697:0:164978:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572486.236706:0:164978:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b3100 x1875444569571712/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572486.236735:0:164978:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571712:192.168.202.157@tcp:36:rm.0 00000100:00100000:3.0:1788572486.236836:0:164978:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990cc0 going to sleep for 50 seconds 00000100:00100000:3.0:1788572486.250695:0:164978:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:164978:1875444569571712:192.168.202.157@tcp:36:rm.0 00000080:00200000:3.0:1788572486.250786:0:164978:0:(vvp_io.c:1826:vvp_io_init()) [0x200027501:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:3.0:1788572486.250801:0:164978:0:(vvp_io.c:304:vvp_io_fini()) [0x200027501:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:3.0:1788572486.250812:0:164978:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x9:0x0](ffff920a50d442c8) 00000100:00100000:2.0:1788572486.619792:0:164868:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a48036600 going to sleep for 25 seconds 00010000:00010000:3.0:1788572487.067756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572488.091862:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572488.668278:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569571968:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572488.668368:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572224:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572488.668395:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572488.668401:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569571840:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572488.668425:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572096:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572488.668460:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572352:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572488.681731:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572224:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572488.685928:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569571840:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572488.698788:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572352:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572488.699578:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569572096:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572488.699652:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569571968:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572489.115781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572490.140752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572491.163871:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572492.189007:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572493.212892:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572493.788705:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569572864:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572493.788792:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569573120:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572493.788823:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:2.0:1788572493.788830:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569572736:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572493.788871:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569572992:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572493.788912:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569573248:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572493.794691:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569572736:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572493.794715:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569572992:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572493.794727:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569573248:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572493.794738:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569572864:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572493.794749:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569573120:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572494.237252:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:2.0:1788572494.321407:0:165055:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 00010000:00010000:3.0:1788572495.259906:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572496.286270:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572497.311074:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572498.331912:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572498.907866:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573504:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572498.908123:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573760:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572498.908161:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:2.0:1788572498.908168:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569573376:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572498.908201:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573632:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572498.908225:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573888:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572498.921946:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573888:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572498.921971:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573760:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572498.924675:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573632:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572498.924699:0:2400:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0000-mdc-ffff920a47b93800 of lustre-MDT0000_UUID@192.168.202.157@tcp abruptly disconnected: reconnecting 00000100:00080000:3.0:1788572498.924706:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572498.924710:0:2400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff920a47b93800: Connection to lustre-MDT0000 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:3.0:1788572498.924717:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572498.924722:0:2400:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9130 00000100:00080000:3.0:1788572498.924735:0:2400:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572498.924751:0:2400:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572498.924792:0:2400:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3800 x1875444569574016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572498.924826:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569573504:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572498.925003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574016:192.168.202.157@tcp:38:ptlrpcd_01_01.0 00000100:00080000:1.0:1788572498.933640:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572498.933661:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572498.933669:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574016:192.168.202.157@tcp:38:ptlrpcd_01_01.0 00010000:00010000:3.0:1788572499.355801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572500.379973:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572501.403811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572502.430953:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572503.451955:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572504.027976:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572504.027984:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572504.027989:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572504.028006:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9257 00000100:00080000:2.0:1788572504.028020:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1788572504.028029:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572504.028044:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572504.028069:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569574272/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572504.028312:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569574144:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572504.028430:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574528:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572504.028475:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572504.028482:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574400:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572504.028510:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572504.028668:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572504.031621:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574656:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572504.031644:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574528:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572504.033511:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569574400:192.168.202.157@tcp:400:kworker.0 00000100:00080000:1.0:1788572504.043115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572504.043138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572504.043144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574272:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572504.476358:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572505.500112:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572506.524218:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572507.550721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572508.574231:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572509.147993:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572509.148003:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572509.148007:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572509.148020:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9262 00000100:00080000:1.0:1788572509.148035:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572509.148044:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572509.148060:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572509.148081:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569574912/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572509.148141:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572509.148146:0:94873:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00080000:1.0:1788572509.148157:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:1.0:1788572509.148160:0:94873:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00100000:1.0:1788572509.148195:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569575040:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572509.148195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572509.148327:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575168:192.168.202.157@tcp:9:kworker.0 00000100:00100000:3.0:1788572509.159436:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575296:192.168.202.157@tcp:9:kworker.0 00000100:00080000:1.0:1788572509.178543:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572509.178565:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572509.178572:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569574912:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572509.187422:0:2396:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff920a48350e00 x1875444569575168/t0(0) o9->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572519 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572509.187444:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788572509.187451:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575168:192.168.202.157@tcp:9:kworker.0 00000100:00080000:1.0:1788572509.206869:0:2396:0:(import.c:2057:ptlrpc_disconnect_idle_interpret()) @@@ inflight=1, refcount=3: rc = 0 req@ffff920a48352d80 x1875444569575296/t0(0) o9->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 0 dl 1788572519 ref 1 fl Interpret:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572509.206882:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to IDLE 00000100:00100000:1.0:1788572509.206887:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575296:192.168.202.157@tcp:9:kworker.0 00010000:00010000:3.0:1788572509.595832:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572510.623004:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572511.643817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572512.668173:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572513.691741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1788572514.211547:0:2400:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572498/real 1788572498] req@ffff920a60ac4a80 x1875444569573376/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572514 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572514.211635:0:2400:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a60ac4a80 x1875444569573376/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572514 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572514.211646:0:2400:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from FULL to DISCONN 00000100:02020000:2.0:1788572514.211651:0:2400:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.157@tcp: Connection to MGS (at 192.168.202.157@tcp) was lost; in progress operations using this service will fail 00000100:00080000:2.0:1788572514.211655:0:2400:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.157@tcp for MGC192.168.202.157@tcp not replayable, auto-deactivating 00000100:00080000:2.0:1788572514.211658:0:2400:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00100000:2.0:1788572514.211662:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a60ac4a80 x1875444569573376/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572514 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572514.211672:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a60ac4380 x1875444569574144/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572520 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572514.211680:0:2400:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a48351880 x1875444569574784/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572525 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:2.0:1788572514.211694:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:2.0:1788572514.211697:0:2400:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:2.0:1788572514.211698:0:2400:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:2.0:1788572514.211701:0:2400:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:2.0:1788572514.211753:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569574144:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572514.211759:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569573376:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572514.212243:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572514.212251:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572514.212255:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572514.212266:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9126 00000100:00080000:3.0:1788572514.212290:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572514.212341:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569575424/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572514.212385:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572514.212390:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572514.212392:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572514.212397:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9267 00000100:00080000:3.0:1788572514.212405:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572514.212411:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572514.212427:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572514.212462:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569575552/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572514.212518:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569575424:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572514.212549:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572514.212571:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569575680:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572514.212716:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575552:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572514.251351:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572514.251438:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572514.251446:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572514.251743:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569575680:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572514.715809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572515.740076:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572516.764260:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572517.787879:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572518.817124:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572519.391228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575808:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572519.401096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572519.401111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572519.401117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569575808:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572519.835749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572520.859856:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572521.883994:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572522.909865:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572523.931675:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572524.507845:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572514/real 1788572514] req@ffff920a529b1500 x1875444569575424/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572524.507891:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569575424/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572524 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572524.507911:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572524.507920:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572524.507925:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569575424:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572524.508193:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572524.508203:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572524.508208:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572524.508218:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9272 00000100:00080000:3.0:1788572524.508232:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572524.508241:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572524.508290:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0a80 x1875444569576064/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572524.508329:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572524.508333:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572524.508335:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572524.508340:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9277 00000100:00080000:3.0:1788572524.508343:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572524.508348:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572524.508361:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572524.508374:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569576192/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572524.508498:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569576320:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572524.508903:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569576064:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572524.508941:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569576192:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572524.513819:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572524.513843:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:1.0:1788572524.513849:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569576192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572524.514032:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569576320:192.168.202.157@tcp:400:kworker.0 00000100:00080000:1.0:1788572524.516314:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) MGC192.168.202.157@tcp: connect to target with instance 0 10000000:01000000:1.0:1788572524.516340:0:2396:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808005 00000100:02000400:1.0:1788572524.516350:0:2396:0:(import.c:1328:ptlrpc_connect_interpret()) Evicted from MGS (at 192.168.202.157@tcp) after server handle changed from 0x265cfe11109bdfc6 to 0x265cfe11109bf70b 00000100:00080000:1.0:1788572524.516357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to EVICTED 00000100:00080000:1.0:1788572524.516365:0:2396:0:(import.c:1718:ptlrpc_import_recovery_state_machine()) evicted from MGS@192.168.202.157@tcp; invalidating 00000100:00100000:1.0:1788572524.516946:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569576064:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572524.517027:0:165278:0:(import.c:1642:ptlrpc_invalidate_import_thread()) thread invalidate import MGC192.168.202.157@tcp to MGS@192.168.202.157@tcp 00000100:00100000:2.0:1788572524.517034:0:165278:0:(import.c:323:ptlrpc_invalidate_import()) Sleeping 20s for inflight to error out 10000000:01000000:2.0:1788572524.517037:0:165278:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808003 00010000:00010000:2.0:1788572524.517059:0:165278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdfd4 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572524.517071:0:165278:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdfd4 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572524.517077:0:165278:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572524.517183:0:165278:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572524.517197:0:165278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79e00/0x8364c713e25bb397 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 4 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdfd4 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572524.517215:0:165278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a51a71800/0x8364c713e25bb3a5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdfe2 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572524.517221:0:165278:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a51a71800/0x8364c713e25bb3a5 lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdfe2 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572524.517226:0:165278:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x0:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572524.522860:0:165278:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572524.522888:0:165278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a51a71800/0x8364c713e25bb3a5 lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdfe2 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572524.522928:0:165278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572524.522935:0:165278:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572524.522946:0:165278:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:2.0:1788572524.527858:0:165278:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572524.527886:0:165278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a47817000/0x8364c713e25bb4fc lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bed56 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572524.527926:0:165278:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a51a71600/0x8364c713e25bb39e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110e400000000 nid: local remote: 0x265cfe11109bdfdb expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:2.0:1788572524.527934:0:165278:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a51a71600/0x8364c713e25bb39e lrc: 3/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x110f400000000 nid: local remote: 0x265cfe11109bdfdb expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572524.527943:0:165278:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x736d61726170:0x3:0x0].0x0 (params) 00010000:00010000:2.0:1788572524.527950:0:165278:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:2.0:1788572524.527953:0:165278:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a51a71600/0x8364c713e25bb39e lrc: 0/0,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 3 type: PLN flags: 0x590f400000000 nid: local remote: 0x265cfe11109bdfdb expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00080000:2.0:1788572524.527976:0:165278:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from EVICTED to RECOVER 00000100:00080000:2.0:1788572524.527982:0:165278:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from RECOVER to FULL 10000000:01000000:2.0:1788572524.527986:0:165278:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808004 00000100:00080000:2.0:1788572524.527990:0:165278:0:(pinger.c:186:ptlrpc_pinger_ir_up()) IR up 00000100:02000000:2.0:1788572524.528000:0:165278:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) MGC192.168.202.157@tcp: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00010000:00010000:2.0:1788572524.956676:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:1.0:1788572525.595767:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:1.0:1788572525.595812:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 341 10000000:01000000:1.0:1788572525.595820:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:1.0:1788572525.595895:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.595912:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.595928:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572525.595970:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576448:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.596110:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572525.599754:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576448:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572525.599814:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.599835:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572525.599958:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576576:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.600109:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572525.623183:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576576:192.168.202.157@tcp:256:ll_cfg_requeue.0 10000000:01000000:1.0:1788572525.623257:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' failed (-2). 00010000:00010000:1.0:1788572525.623272:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.623286:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00000400:1.0:1788572525.623298:0:160932:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.202.157@tcp: IR log lustre-cliir failed, not fatal: rc = -2 10000000:01000000:1.0:1788572525.636308:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-client 10000000:01000000:1.0:1788572525.636320:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-client-ffff920a47b93800 from 52 10000000:01000000:1.0:1788572525.636539:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-client (res 0x65727473756c) 00010000:00010000:1.0:1788572525.636675:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.636691:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.636706:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572525.636746:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576704:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.639030:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572525.645504:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576704:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:1.0:1788572525.645549:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf719 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572525.645566:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bf719 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572525.645704:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576832:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.645830:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572525.696057:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576832:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.696161:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576960:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000100:00100000:1.0:1788572525.696296:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7480 going to sleep for 11 seconds 00000100:00100000:1.0:1788572525.707860:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569576960:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572525.715131:0:165279:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x3:0x0] index 64768 count 51 00000020:01000000:2.0:1788572525.722871:0:160932:0:(obd_config.c:2145:class_config_parse_llog()) Processed log lustre-client gen 52-51 (rc=0) 10000000:01000000:2.0:1788572525.723046:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-client' succeeded (0). 00010000:00010000:2.0:1788572525.723064:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf719 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.723079:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a60a78e00/0x8364c713e25bb645 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf719 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572525.723097:0:160932:0:(mgc_request.c:611:do_requeue()) updating log params 10000000:01000000:2.0:1788572525.723101:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log params-ffff920a47b93800 from 7 10000000:01000000:2.0:1788572525.723104:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for params (res 0x736d61726170) 00010000:00010000:2.0:1788572525.723162:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.723172:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.723255:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 3/1,0 mode: --/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572525.723494:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577088:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.724520:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572525.736241:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577088:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572525.736300:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf720 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.736316:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 4/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bf720 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572525.736443:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577216:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.736567:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572525.746999:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577216:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.747099:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577344:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.747240:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572525.753732:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577344:192.168.202.157@tcp:503:ll_cfg_requeue.0 00000040:00080000:1.0:1788572525.755485:0:165280:0:(llog.c:862:llog_process_thread()) stop processing plain [0xa:0x2:0x0] index 64768 count 7 00000020:01000000:2.0:1788572525.756082:0:160932:0:(obd_config.c:2145:class_config_parse_llog()) Processed log params gen 7-6 (rc=0) 10000000:01000000:2.0:1788572525.756108:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'params' succeeded (0). 00010000:00010000:2.0:1788572525.756123:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 3/1,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf720 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.756138:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a47817c00/0x8364c713e25bb64c lrc: 2/0,0 mode: CR/CR res: [0x736d61726170:0x3:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf720 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:2.0:1788572525.756150:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-sptlrpc 10000000:01000000:2.0:1788572525.756153:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-sptlrpc-ffff920a44393380 from 1 10000000:01000000:2.0:1788572525.756157:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-sptlrpc (res 0x65727473756c) 00010000:00010000:2.0:1788572525.756210:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.756219:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.756232:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572525.756267:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577472:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.756394:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572525.764304:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577472:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572525.764347:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf727 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.764368:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bf727 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572525.764462:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577600:192.168.202.157@tcp:501:ll_cfg_requeue.0 00000100:00100000:2.0:1788572525.764545:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572525.775675:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577600:192.168.202.157@tcp:501:ll_cfg_requeue.0 10000000:01000000:2.0:1788572525.775746:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-sptlrpc' failed (-2). 00010000:00010000:2.0:1788572525.775761:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf727 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572525.775790:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a47816a00/0x8364c713e25bb653 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 3 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf727 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572525.980641:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572527.003874:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572527.700728:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572527.705708:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584468352 00000100:00100000:2.0:1788572527.705732:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584468352 00000100:00100000:2.0:1788572527.705841:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 9 00000100:00100000:2.0:1788572527.705855:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a529b2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f79285ed-4685-45a8-8731-5e2522a143f3+3:201334:x1875444584468352:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572527.705880:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572527.706147:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572527.706156:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a529b2680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:f79285ed-4685-45a8-8731-5e2522a143f3+3:201334:x1875444584468352:12345-192.168.202.157@tcp:104: Request processed in 303us (5446us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572527.706164:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 9 00010000:00010000:1.0:1788572527.713492:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572527.713513:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.713688:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79a00 already unused, calling callback (ffffffffc1336c20) 10000000:00010000:1.0:1788572527.713694:0:160930:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.713707:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572527.713716:0:160930:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572527.713722:0:160930:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572527.713819:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.713826:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572527.713856:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.713865:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a60a79a00/0x8364c713e25bb63e lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf712 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572527.713899:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.713902:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572527.713904:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572528.032289:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 10000000:01000000:2.0:1788572528.731757:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788572528.731780:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 341 10000000:01000000:2.0:1788572528.731789:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572528.731869:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572528.731882:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572528.731895:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572528.731932:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577856:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572528.732053:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572528.743972:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577856:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572528.744027:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572528.744043:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572528.744151:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577984:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788572528.744277:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8dc00 going to sleep for 11 seconds 00000100:00100000:2.0:1788572528.759664:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569577984:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00080000:2.0:1788572528.759747:0:160932:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a4549f800 already not connected (conn 15, was 0): DISCONN 00010000:00080000:2.0:1788572528.759784:0:160932:0:(ldlm_lib.c:85:import_set_conn()) imp ffff920a4549f800@lustre-MDT0000-mdc-ffff920a47b93800: found existing conn 192.168.202.157@tcp, moved to head 00000100:00080000:2.0:1788572528.759793:0:160932:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572528.759799:0:160932:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:2.0:1788572528.759811:0:160932:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572528.759827:0:160932:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572528.759866:0:160932:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569578112/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:2.0:1788572528.760076:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572528.760089:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572528.760101:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572528.762495:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569578112:192.168.202.157@tcp:38:ll_cfg_requeue.0 00010000:00080000:1.0:1788572528.777450:0:2396:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=0) req@ffff920a60ac4380 x1875444569578112/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/416 e 0 to 0 dl 1788572538 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'ll_cfg_requeue.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572528.777482:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-MDT0000-mdc-ffff920a47b93800: connect to target with instance 82 00000100:00080000:1.0:1788572528.777511:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to REPLAY 00000100:00080000:1.0:1788572528.777524:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572528.777527:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350988 gen 1 00000100:00100000:1.0:1788572528.777535:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a48352a00 x1875444569533312/t347892350990(347892350990) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572418 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572528.777544:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350988 last 0 00000100:00080000:1.0:1788572528.777554:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a48352a00 x1875444569533312/t347892350990(347892350990) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572418 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572528.777568:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569578112:192.168.202.157@tcp:38:ll_cfg_requeue.0 00000100:00100000:1.0:1788572528.777647:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569533312:192.168.202.157@tcp:36:mcreate.0 00010000:00010000:1.0:1788572529.052262:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572529.627912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level REPLAY/5 force 0 force_next 1 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572529.627919:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: REPLAY) 00000100:00100000:3.0:1788572529.628071:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569578240:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572529.628200:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572529.628207:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569578368:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572529.636448:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569578368:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572529.636464:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569578240:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572530.075804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572531.099824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572532.128235:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572533.148193:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572533.326401:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a48352a00 x1875444569533312/t347892350990(347892350990) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572558 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.326422:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.326426:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350988 gen 1 00000100:00100000:1.0:1788572533.326432:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a48352a00 x1875444569533312/t347892350990(347892350990) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572558 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.326441:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350988 last 347892350990 00000100:00080000:1.0:1788572533.326450:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3dc00 x1875444569533824/t347892350991(347892350991) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572418 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.326463:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569533312:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:1.0:1788572533.326493:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569533824:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572533.345358:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350990 gen 1 00000100:00100000:1.0:1788572533.345405:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3dc00 x1875444569533824/t347892350991(347892350991) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 2 fl Rpc:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.345423:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a51b3dc00 x1875444569533824/t347892350991(347892350991) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.345433:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.345436:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350990 gen 1 00000100:00100000:1.0:1788572533.345440:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3dc00 x1875444569533824/t347892350991(347892350991) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.345448:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350990 last 347892350991 00000100:00080000:1.0:1788572533.345457:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3e680 x1875444569535360/t347892350992(347892350992) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572465 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.345471:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3dc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569533824:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572533.345499:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569535360:192.168.202.157@tcp:36:mcreate.0 00000100:00080000:3.0:1788572533.366723:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a51b3e680 x1875444569535360/t347892350992(347892350992) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572533.366753:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:3.0:1788572533.366757:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350990 gen 1 00000100:00100000:3.0:1788572533.366764:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a51b3dc00 x1875444569533824/t347892350991(347892350991) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572533.366773:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350990 last 347892350992 00000100:00080000:3.0:1788572533.366784:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a51b3f800 x1875444569535872/t347892350993(347892350993) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572465 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572533.366799:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51b3e680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569535360:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:3.0:1788572533.366826:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51b3f800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569535872:192.168.202.157@tcp:36:rm.0 00000100:00080000:1.0:1788572533.465504:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a48350000 x1875444569566464/t347892351007(347892351007) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 568/448 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.465524:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.465527:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350993 gen 1 00000100:00100000:1.0:1788572533.465535:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a45b36d80 x1875444569564672/t347892351006(347892351006) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.465543:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350993 last 347892351007 00000100:00080000:1.0:1788572533.465551:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a529b0000 x1875444569568512/t347892351010(347892351010) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572492 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.465564:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569566464:192.168.202.157@tcp:36:lfs.0 00000100:00100000:1.0:1788572533.465633:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569568512:192.168.202.157@tcp:36:mcreate.0 00000100:00080000:1.0:1788572533.484714:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a529b0000 x1875444569568512/t347892351010(347892351010) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.484733:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.484736:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350993 gen 1 00000100:00100000:1.0:1788572533.484756:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a45b36d80 x1875444569564672/t347892351006(347892351006) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.484765:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350993 last 347892351010 00000100:00080000:1.0:1788572533.484774:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a48352680 x1875444569569024/t347892351011(347892351011) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572493 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.484787:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569568512:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:1.0:1788572533.484814:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569569024:192.168.202.157@tcp:36:rm.0 00000100:00080000:1.0:1788572533.488218:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a48352680 x1875444569569024/t347892351011(347892351011) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.488237:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.488240:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350993 gen 1 00000100:00100000:1.0:1788572533.488248:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a45b36d80 x1875444569564672/t347892351006(347892351006) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.488256:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350993 last 347892351011 00000100:00080000:1.0:1788572533.488264:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a529b1f80 x1875444569571200/t347892351012(347892351012) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572541 ref 1 fl New:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.488277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569569024:192.168.202.157@tcp:36:rm.0 00000100:00100000:1.0:1788572533.488314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569571200:192.168.202.157@tcp:36:mcreate.0 00000100:00080000:1.0:1788572533.492437:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a529b1f80 x1875444569571200/t347892351012(347892351012) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 528/448 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.492454:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.492457:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350993 gen 1 00000100:00100000:1.0:1788572533.492464:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a45b36d80 x1875444569564672/t347892351006(347892351006) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.492473:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350993 last 347892351012 00000100:00080000:1.0:1788572533.492482:0:2396:0:(client.c:3554:ptlrpc_replay_req()) @@@ REPLAY req@ffff920a529b3100 x1875444569571712/t347892351013(347892351013) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572541 ref 1 fl New:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572533.492494:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569571200:192.168.202.157@tcp:36:mcreate.0 00000100:00100000:1.0:1788572533.492519:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569571712:192.168.202.157@tcp:36:rm.0 00000100:00080000:1.0:1788572533.513106:0:2396:0:(client.c:3439:ptlrpc_replay_interpret()) @@@ got reply req@ffff920a529b3100 x1875444569571712/t347892351013(347892351013) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 2 fl Interpret:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.513127:0:2396:0:(import.c:1744:ptlrpc_import_recovery_state_machine()) replay requested by lustre-MDT0000_UUID 00000100:00100000:1.0:1788572533.513130:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 347892350993 gen 1 00000100:00100000:1.0:1788572533.513138:0:2396:0:(client.c:3170:ptlrpc_free_committed()) @@@ stopping search req@ffff920a45b36d80 x1875444569564672/t347892351006(347892351006) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/456 e 0 to 0 dl 1788572563 ref 1 fl Complete:RQU/204/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572533.513147:0:2396:0:(recover.c:53:ptlrpc_replay_next()) import ffff920a4549f800 from lustre-MDT0000_UUID committed 347892350993 last 347892351013 00000100:00080000:1.0:1788572533.513153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from REPLAY to REPLAY_LOCKS 00010000:00080000:1.0:1788572533.513705:0:165336:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0000-mdc-ffff920a47b93800 to lustre-MDT0000_UUID@192.168.202.157@tcp 00010000:00010000:1.0:1788572533.513710:0:165336:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0000-mdc-ffff920a47b93800 (1) 00010000:00010000:1.0:1788572533.513792:0:165336:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb5f8 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000000 nid: local remote: 0x265cfe11109bf5c9 expref: -99 pid: 164868 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572533.513813:0:165336:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0000-mdc-ffff920a47b93800 00000100:00080000:1.0:1788572533.514923:0:165336:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00000100:02000000:1.0:1788572533.532240:0:2396:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) lustre-MDT0000-mdc-ffff920a47b93800: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00100000:1.0:1788572533.532251:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569578496:192.168.202.157@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572533.660007:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569578752:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572533.660155:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:3.0:1788572533.660163:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569578624:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572533.695032:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569578624:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572533.695069:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569578752:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572534.175170:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572535.196855:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572536.222468:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572537.255704:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:2.0:1788572538.177235:0:165380:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572538.177255:0:165380:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572538.177300:0:165380:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572538.177384:0:165380:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 165380 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.177396:0:165380:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 165380 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.177409:0:165380:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 165380 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572538.177446:0:165380:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569578880:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572538.177567:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4900 going to sleep for 25 seconds 00000100:00100000:2.0:1788572538.200120:0:165380:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569578880:192.168.202.157@tcp:101:lfs.0 00010000:00010000:2.0:1788572538.200160:0:165380:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200177:0:165380:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200191:0:165380:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200204:0:165380:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200220:0:165380:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200237:0:165380:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200245:0:165380:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572538.200258:0:165380:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac4a80 x1875444569578880/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572568 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572538.200533:0:165380:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572538.200875:0:165380:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:2.0:1788572538.200887:0:165380:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572538.202786:0:165380:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572538.202812:0:165380:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:3.0:1788572538.202823:0:165380:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572538.202827:0:165380:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7302bcfcb8 00010000:00010000:3.0:1788572538.202835:0:165380:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572538.202850:0:165380:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:3.0:1788572538.208002:0:165380:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572538.208023:0:165380:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572538.208058:0:165380:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572538.208084:0:165380:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572538.208106:0:165380:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572538.208121:0:165380:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572538.208131:0:165380:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572538.208135:0:165380:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:3.0:1788572538.208144:0:165380:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7302bcfcb8 00010000:00010000:3.0:1788572538.208148:0:165380:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572538.208158:0:165380:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572538.211409:0:165380:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572538.211454:0:165380:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572538.211822:0:165380:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572538.211842:0:165380:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572538.211851:0:165380:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572538.211862:0:165380:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572538.211866:0:165380:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:3.0:1788572538.211875:0:165380:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7302bcfbb0 00010000:00010000:3.0:1788572538.211880:0:165380:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572538.211890:0:165380:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572538.213059:0:165380:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572538.213086:0:165380:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:3.0:1788572538.214385:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:3.0:1788572538.214414:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43167c00 uarg=0000000000000000 00800000:00000004:3.0:1788572538.214425:0:165380:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9244, max_age 9295 00000100:00100000:3.0:1788572538.214482:0:165380:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569579008:192.168.202.157@tcp:41:lfs.0 00000100:00100000:3.0:1788572538.216755:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990780 going to sleep for 11 seconds 00000100:00100000:1.0:1788572538.240076:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52847e40 going to sleep for 25 seconds 00000100:00100000:1.0:1788572538.246648:0:165380:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36d80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569579136:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572538.246809:0:165380:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317318/353529 objects 1023722/1024000 00000080:00200080:1.0:1788572538.247061:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247092:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247102:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247107:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247111:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247115:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247120:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247124:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247128:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247131:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247135:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247139:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247143:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247147:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247151:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247154:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247158:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247162:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247166:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247170:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247174:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247177:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247181:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247185:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247189:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247192:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247198:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247202:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247207:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247210:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247215:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247219:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247224:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247228:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247232:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247236:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247239:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247242:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247246:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247249:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247253:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247257:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247261:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247264:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247269:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247288:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247295:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247300:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247305:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247309:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247313:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247318:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247323:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247326:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247330:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247334:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247338:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247342:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247346:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247349:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247354:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247357:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247362:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247366:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247370:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247373:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247377:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247381:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247385:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247389:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247393:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247397:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247401:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247404:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247409:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247412:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247417:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247420:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247424:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247427:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247431:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247435:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247438:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247442:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247446:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247449:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247453:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247457:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247462:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247466:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247469:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247473:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247477:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247480:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247484:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247488:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247492:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247495:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247498:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247502:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247505:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247509:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247513:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247516:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247520:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247524:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247527:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247530:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247534:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247538:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247542:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247545:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247549:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247552:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247556:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247559:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247562:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247565:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247569:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247572:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247576:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247580:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247627:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247633:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247637:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00800000:00000080:1.0:1788572538.247641:0:165380:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.247645:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.247653:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00020000:00000004:1.0:1788572538.247671:0:165380:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9244, max_age 9295 00000100:00080000:1.0:1788572538.247689:0:165380:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9214 00000100:00080000:1.0:1788572538.247705:0:165380:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572538.247719:0:165380:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572538.247764:0:165380:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350e00 x1875444569579264/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572538.247825:0:165380:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a48352a00 x1875444569579392/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572538.247836:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7240 going to sleep for 0 seconds 00000100:00100000:2.0:1788572538.248452:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569579264:192.168.202.157@tcp:8:lfs.0 00000100:00100000:2.0:1788572538.258288:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a47b93800: committing for last_committed 38654705666 gen 4 00000100:00080000:2.0:1788572538.258313:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff920a47b93800: connect to target with instance 9 00000100:00080000:2.0:1788572538.258326:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572538.258341:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572538.258347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788572538.258359:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572538.258367:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788572538.258386:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 00000000a795402a): req@ffff920a48352a00 x1875444569579392/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572538.258418:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569579264:192.168.202.157@tcp:8:lfs.0 00000100:00100000:1.0:1788572538.260123:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572538.271524:0:165380:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569579392:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572538.271564:0:165380:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261721/262144 00000080:00200080:1.0:1788572538.271714:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.271747:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00020000:00000004:1.0:1788572538.271757:0:165380:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9244, max_age 9295 00000100:00080000:1.0:1788572538.271771:0:165380:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9214 00000100:00080000:1.0:1788572538.271784:0:165380:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572538.271797:0:165380:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572538.271827:0:165380:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569579520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572538.271867:0:165380:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a48352a00 x1875444569579648/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572538.271883:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7240 going to sleep for 0 seconds 00010000:00010000:2.0:1788572538.278840:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572538.285544:0:165380:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a528a7240 going to sleep for 11 seconds 00000100:00100000:1.0:1788572538.290909:0:165380:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165380:1875444569579648:192.168.202.157@tcp:13:lfs.0 00020000:00000004:1.0:1788572538.290958:0:165380:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261737/262144 00000080:00200080:1.0:1788572538.291054:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291077:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291085:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291090:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291094:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291097:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291101:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291104:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291108:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291111:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291114:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291117:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291120:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291124:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291127:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291130:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291134:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291137:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291141:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291144:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291148:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291151:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291155:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291158:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291162:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291165:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291169:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291172:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291175:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291179:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291183:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291186:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291190:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291193:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291197:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291200:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291204:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291208:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291212:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291215:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291219:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291222:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291226:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291230:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291235:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291239:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291243:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291246:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291250:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291254:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291257:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291260:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291263:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291266:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291269:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291402:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291411:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291416:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291420:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291423:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291427:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291431:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291435:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291438:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291442:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291445:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291448:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291452:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291456:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291459:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291463:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291467:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291470:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291474:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291478:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291482:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291486:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291489:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291493:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291496:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291500:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291504:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291508:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291511:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291515:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291519:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291523:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291527:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291532:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291535:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291539:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291542:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291545:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291549:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291553:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291557:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291561:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291565:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291569:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291573:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291577:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291581:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291646:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291652:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291657:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291660:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291664:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291667:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291671:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291674:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291678:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291682:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291686:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291689:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291693:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291697:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291701:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291704:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291708:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291711:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291716:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291720:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291734:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291739:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200080:1.0:1788572538.291743:0:165380:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fff3cca52b0 00020000:00000080:1.0:1788572538.291747:0:165380:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a46758400 uarg=0000000000000000 00000080:00200008:1.0:1788572538.291765:0:165380:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:1.0:1788572539.308654:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572540.316166:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572541.339895:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572542.363777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572543.392686:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572543.968874:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580544:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.969002:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.969042:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:1.0:1788572543.969049:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569580416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.969084:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580672:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.969109:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.977063:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569580416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.977094:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.977107:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580544:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.977119:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572543.988554:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569580672:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572544.413894:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572546.461162:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572547.491706:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572548.507759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572548.516199:0:165809:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec 00000100:00100000:2.0:1788572549.084021:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569581056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572549.084147:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581312:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572549.084174:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581568:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572549.084197:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:2.0:1788572549.084200:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572549.084213:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581440:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572549.119460:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569581056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572549.119495:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581312:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572549.119510:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581568:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572549.119553:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572549.119565:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569581440:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572549.532084:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572550.556214:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200008:1.0:1788572551.107211:0:165848:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1788572551.107230:0:165848:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572551.107259:0:165848:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572551.107283:0:165848:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572551.107302:0:165848:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572551.107312:0:165848:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572551.107322:0:165848:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572551.107326:0:165848:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572551.107335:0:165848:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7300c8fcb8 00010000:00010000:1.0:1788572551.107338:0:165848:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572551.107349:0:165848:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:1.0:1788572551.108576:0:165848:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572551.113988:0:165848:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572551.114046:0:165848:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572551.114073:0:165848:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572551.114094:0:165848:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572551.114114:0:165848:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572551.114128:0:165848:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572551.114132:0:165848:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572551.114143:0:165848:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7300c8fcb8 00010000:00010000:1.0:1788572551.114147:0:165848:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572551.114157:0:165848:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572551.114441:0:165848:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572551.114453:0:165848:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572551.114460:0:165848:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572551.114469:0:165848:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572551.114475:0:165848:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572551.114479:0:165848:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572551.114482:0:165848:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572551.114486:0:165848:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7300c8fbb0 00010000:00010000:1.0:1788572551.114489:0:165848:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572551.114496:0:165848:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:1.0:1788572551.114506:0:165848:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:1.0:1788572551.114518:0:165848:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:1.0:1788572551.115367:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:1.0:1788572551.115393:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b400 uarg=0000000000000000 00800000:00000004:1.0:1788572551.115403:0:165848:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9296, max_age 9308 00000100:00100000:1.0:1788572551.115464:0:165848:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581696:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572551.115605:0:165848:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d9c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572551.122992:0:165848:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581696:192.168.202.157@tcp:41:lfs.0 00800000:00000004:1.0:1788572551.123044:0:165848:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317221/353529 objects 1023634/1024000 00000080:00200080:1.0:1788572551.123196:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:1.0:1788572551.123219:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675b400 uarg=0000000000000000 00800000:00000004:1.0:1788572551.123229:0:165848:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9296, max_age 9308 00000100:00100000:1.0:1788572551.123278:0:165848:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581824:192.168.202.157@tcp:41:lfs.0 00000100:00100000:1.0:1788572551.123372:0:165848:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52b8d9c0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572551.125657:0:165848:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581824:192.168.202.157@tcp:41:lfs.0 00800000:00000004:3.0:1788572551.125700:0:165848:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317318/353529 objects 1023722/1024000 00000080:00200080:3.0:1788572551.125765:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125785:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125794:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125798:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125803:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125817:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125822:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125826:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125831:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125834:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125839:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125842:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125847:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125850:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125855:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125859:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125863:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125866:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125878:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125883:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125888:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125892:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125896:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125899:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125904:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125908:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125912:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125917:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125921:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125924:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125927:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125931:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125935:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125939:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125942:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125955:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125960:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125964:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125968:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125972:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125976:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125979:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125983:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125987:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125991:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.125995:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.125999:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126003:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126007:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126010:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126014:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126018:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126023:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126026:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126178:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126185:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126189:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126193:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126197:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126201:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126206:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126210:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126214:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126218:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126222:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126225:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126239:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126244:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126249:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126253:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126257:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126261:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126265:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126269:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126273:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126276:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126280:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126284:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126289:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126292:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126296:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126300:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126304:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126308:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126312:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126315:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126319:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126322:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126326:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126330:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126334:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126337:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126341:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126345:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126349:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126352:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126356:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126360:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126363:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126367:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126371:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126375:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126379:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126383:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126387:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126391:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126395:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126399:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126403:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126406:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126410:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126414:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126418:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126422:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126426:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126429:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126433:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126437:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126441:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126445:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126449:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126453:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126457:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126460:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126464:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00800000:00000080:3.0:1788572551.126468:0:165848:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.126472:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.126480:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00020000:00000004:3.0:1788572551.126489:0:165848:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9296, max_age 9308 00000100:00100000:3.0:1788572551.126536:0:165848:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581952:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572551.126656:0:165848:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572551.130785:0:165848:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569581952:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572551.130834:0:165848:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261721/262144 00000080:00200080:3.0:1788572551.130911:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.130934:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00020000:00000004:3.0:1788572551.130943:0:165848:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9296, max_age 9308 00000100:00100000:3.0:1788572551.131139:0:165848:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569582080:192.168.202.157@tcp:13:lfs.0 00000100:00100000:3.0:1788572551.131269:0:165848:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51865300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572551.135446:0:165848:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:165848:1875444569582080:192.168.202.157@tcp:13:lfs.0 00020000:00000004:3.0:1788572551.135487:0:165848:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261737/262144 00000080:00200080:3.0:1788572551.135684:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135708:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135716:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135720:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135725:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135728:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135733:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135737:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135740:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135744:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135748:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135754:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135757:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135760:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135764:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135767:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135771:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135775:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135778:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135782:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135786:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135790:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135794:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135797:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135802:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135816:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135820:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135824:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135828:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135831:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.135835:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.135838:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136014:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136023:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136098:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136105:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136110:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136114:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136117:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136120:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136124:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136127:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136131:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136134:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136139:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136142:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136146:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136149:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136153:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136157:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136161:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136164:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136168:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136172:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136175:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136179:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136183:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136187:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136190:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136193:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136196:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136200:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136203:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136207:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136211:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136214:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136217:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136221:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136225:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136228:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136232:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136236:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136239:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136242:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136246:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136249:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136252:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136255:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136258:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136262:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136265:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136268:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136272:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136275:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136278:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136282:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136287:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136290:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136294:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136297:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136301:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136304:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136308:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136311:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136315:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136318:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136321:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136325:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136329:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136332:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136336:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136339:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136343:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136346:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136350:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136353:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136357:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136361:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136364:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136367:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136371:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136375:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136379:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136382:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136386:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136390:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136393:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136396:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136400:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136403:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136408:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136411:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136416:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136419:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200080:3.0:1788572551.136423:0:165848:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffcbd23fc90 00020000:00000080:3.0:1788572551.136427:0:165848:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a43165400 uarg=0000000000000000 00000080:00200008:3.0:1788572551.136446:0:165848:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:3.0:1788572552.604566:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572554.204688:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569582208:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.204823:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.204858:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582720:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.204892:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [1->0] 00000100:00100000:1.0:1788572554.204899:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582336:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.204923:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582592:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.207817:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582336:192.168.202.157@tcp:400:kworker.0 00000100:00080000:1.0:1788572554.207846:0:2397:0:(recover.c:205:ptlrpc_request_handle_notconn()) import lustre-MDT0001-mdc-ffff920a47b93800 of lustre-MDT0001_UUID@192.168.202.157@tcp abruptly disconnected: reconnecting 00000100:00080000:1.0:1788572554.207854:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:1.0:1788572554.207859:0:2397:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff920a47b93800: Connection to lustre-MDT0001 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:1.0:1788572554.207867:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572554.207874:0:2397:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9206 00000100:00080000:1.0:1788572554.207913:0:2397:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572554.207928:0:2397:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572554.207967:0:2397:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569582848/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572554.208304:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.209646:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569582208:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572554.211305:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582592:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572554.211654:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572554.211662:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572554.211666:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569582848:192.168.202.157@tcp:38:ptlrpcd_00_00.0 00000100:00100000:1.0:1788572554.212083:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569582720:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572554.651860:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572556.699762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572558.748900:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572559.324967:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569582976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.325872:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572559.325887:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569583104:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.327394:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569582976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.328905:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569583104:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572559.329539:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569583232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572559.331265:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569583488:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.331391:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572559.331400:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569583360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.339002:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569583360:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572559.339027:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569583488:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572559.339127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572559.339135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572559.339140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569583232:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572560.795782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572562.843751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572564.444073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569583872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572564.444653:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569583616:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.444769:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569584000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.444812:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:1.0:1788572564.444822:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569583744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.444861:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569584128:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572564.464739:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572564.464760:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:3.0:1788572564.464767:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569583872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572564.473235:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569584000:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.473290:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569583744:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.473306:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569584128:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572564.473319:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569583616:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572564.894664:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572566.947831:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572568.988337:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572569.569190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569584512:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572569.578183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572569.578210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-19) 00000100:00100000:2.0:1788572569.578217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569584512:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572571.035858:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572573.086100:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572574.683964:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572574.683972:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572574.683976:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572574.683987:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9328 00000100:00080000:3.0:1788572574.683997:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572574.684014:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572574.684027:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572574.684044:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569585152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572574.684079:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572574.684083:0:103578:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0000-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00080000:3.0:1788572574.684091:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from FULL to CONNECTING 00000100:00080000:3.0:1788572574.684094:0:103578:0:(import.c:2147:ptlrpc_disconnect_and_idle_import()) lustre-OST0001-osc-ffff920a47b93800: disconnect after 23s idle 00000100:00100000:3.0:1788572574.684123:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569585024:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572574.684226:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:3.0:1788572574.684233:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569584896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572574.696948:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569584896:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572574.696974:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569585024:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572575.135165:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572577.180419:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1788572579.227824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572579.803793:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572579.803800:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572579.803803:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572579.803821:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9333 00000100:00080000:3.0:1788572579.803829:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572579.803836:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572579.803847:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572579.803860:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569585792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572579.804688:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569585536:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572579.804794:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572579.804802:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569585664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572579.807047:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569585536:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572579.812262:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569585664:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572580.381023:0:2392:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:2.0:1788572580.382708:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584503168 00000100:00100000:2.0:1788572580.382723:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584503168 00000100:00100000:2.0:1788572580.382802:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 10 00000100:00100000:2.0:1788572580.382814:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a529b0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:201334:x1875444584503168:12345-192.168.202.157@tcp:104: 00010000:00010000:2.0:1788572580.382835:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572580.382928:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:2.0:1788572580.382937:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a529b0a80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:f79285ed-4685-45a8-8731-5e2522a143f3+3:201334:x1875444584503168:12345-192.168.202.157@tcp:104: Request processed in 125us (1926us total) trans 0 rc 0/0 00000100:00100000:2.0:1788572580.382945:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 10 00010000:00010000:1.0:1788572580.383087:0:160929:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572580.383096:0:160929:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400000000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.383109:0:160929:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816400 already unused, calling callback (ffffffffc1336c20) 10000000:00010000:1.0:1788572580.383113:0:160929:0:(mgc_request.c:922:mgc_blocking_ast()) ### MGC blocking CB ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1400400000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.383121:0:160929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1408400000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:00010000:1.0:1788572580.383129:0:160929:0:(mgc_request.c:928:mgc_blocking_ast()) ### MGC cancel CB ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1409400000000 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 10000000:01000000:1.0:1788572580.383135:0:160929:0:(mgc_request.c:932:mgc_blocking_ast()) Lock res [0x65727473756c:0x2:0x0].0x0 (lustre) 00010000:00010000:1.0:1788572580.383196:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 2/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.383204:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572580.383239:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 1/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.383247:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: MGC192.168.202.157@tcp lock: ffff920a47816400/0x8364c713e25bb65a lrc: 0/0,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x5c09400000020 nid: local remote: 0x265cfe11109bf846 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572580.383266:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.383268:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572580.383270:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572580.383277:0:2397:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1788572580.383291:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569585920:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1788572580.394028:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569585920:192.168.202.157@tcp:103:ldlm_bl.0 10000000:01000000:2.0:1788572581.723794:0:160932:0:(mgc_request.c:611:do_requeue()) updating log lustre-cliir 10000000:01000000:2.0:1788572581.723803:0:160932:0:(mgc_request.c:1802:mgc_process_log()) Process log lustre-cliir-ffff920a47b93800 from 344 10000000:01000000:2.0:1788572581.723809:0:160932:0:(mgc_request.c:996:mgc_enqueue()) Enqueue for lustre-cliir (res 0x65727473756c) 00010000:00010000:2.0:1788572581.723860:0:160932:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x10000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572581.723872:0:160932:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000000000000 ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x0 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572581.723884:0:160932:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 3/1,0 mode: --/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x0 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572581.723912:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569586048:192.168.202.157@tcp:101:ll_cfg_requeue.0 00000100:00100000:2.0:1788572581.724009:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572581.731268:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569586048:192.168.202.157@tcp:101:ll_cfg_requeue.0 00010000:00010000:2.0:1788572581.731298:0:160932:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bfcc9 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572581.731316:0:160932:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 4/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x1000000000000 nid: local remote: 0x265cfe11109bfcc9 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572581.731397:0:160932:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569586176:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00100000:2.0:1788572581.731484:0:160932:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572581.753320:0:160932:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ll_cfg_requeue:f79285ed-4685-45a8-8731-5e2522a143f3:160932:1875444569586176:192.168.202.157@tcp:256:ll_cfg_requeue.0 00000100:00080000:2.0:1788572581.753370:0:160932:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a51ac4800 already not connected (conn 13, was 0): DISCONN 00010000:00080000:2.0:1788572581.753383:0:160932:0:(ldlm_lib.c:85:import_set_conn()) imp ffff920a51ac4800@lustre-MDT0001-mdc-ffff920a47b93800: found existing conn 192.168.202.157@tcp, moved to head 00000100:00080000:2.0:1788572581.753388:0:160932:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572581.753393:0:160932:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 0 00000100:00080000:2.0:1788572581.753401:0:160932:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572581.753411:0:160932:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572581.753436:0:160932:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569586304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 10000000:01000000:2.0:1788572581.753499:0:160932:0:(mgc_request.c:1879:mgc_process_log()) MGC192.168.202.157@tcp: configuration from log 'lustre-cliir' succeeded (0). 00010000:00010000:2.0:1788572581.753511:0:160932:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 3/1,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bfcc9 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572581.753521:0:160932:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: MGC192.168.202.157@tcp lock: ffff920a47816200/0x8364c713e25bb668 lrc: 2/0,0 mode: CR/CR res: [0x65727473756c:0x2:0x0].0x0 rrc: 2 type: PLN flags: 0x11000000000000 nid: local remote: 0x265cfe11109bfcc9 expref: -99 pid: 160932 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00080000:3.0:1788572581.777662:0:166134:0:(ldlm_request.c:2972:ldlm_lock_replay_thread()) lock replay thread lustre-MDT0001-mdc-ffff920a47b93800 to lustre-MDT0001_UUID@192.168.202.157@tcp 00010000:00010000:3.0:1788572581.777669:0:166134:0:(ldlm_request.c:2888:ldlm_cancel_unused_locks_for_replay()) Dropping as many unused locks as possible before replay for namespace lustre-MDT0001-mdc-ffff920a47b93800 (0) 00010000:00010000:3.0:1788572581.777675:0:166134:0:(ldlm_request.c:2901:ldlm_cancel_unused_locks_for_replay()) Canceled 0 unused locks from namespace lustre-MDT0001-mdc-ffff920a47b93800 00000100:00080000:3.0:1788572581.778266:0:166134:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from REPLAY_LOCKS to REPLAY_WAIT 00010000:00010000:1.0:1788572582.300066:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572584.349550:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572584.928628:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level REPLAY_WAIT/7 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572584.928635:0:103578:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: REPLAY_WAIT) 00000100:00100000:2.0:1788572584.928667:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569586560:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.928757:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:2.0:1788572584.928764:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569586688:192.168.202.157@tcp:400:kworker.0 00010000:00010000:2.0:1788572584.935775:0:2399:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 0/0 locks to regular blp list, 1 blwis in pool 00000100:00100000:2.0:1788572584.935796:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569586688:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572584.935811:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569586560:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572585.371833:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:02000000:2.0:1788572585.527836:0:2396:0:(import.c:1787:ptlrpc_import_recovery_state_machine()) lustre-MDT0001-mdc-ffff920a47b93800: Connection restored to 192.168.202.157@tcp (at 192.168.202.157@tcp) 00000100:00100000:2.0:1788572585.527843:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569586432:192.168.202.157@tcp:400:ldlm_lock_repla.0 00000100:00100000:3.0:1788572586.972215:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569586944:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572586.972305:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 2 async RPCs [0->1] 00000100:00100000:3.0:1788572586.972312:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569586816:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572586.972354:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569587072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572586.975134:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569586816:192.168.202.157@tcp:400:kworker.0 00010000:00080000:3.0:1788572586.975158:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a529b3800 x1875444569587072/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572602 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572586.975170:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569587072:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572586.975182:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569586944:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572587.419817:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572589.470886:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200080:1.0:1788572591.196911:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.196954:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.196969:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.196975:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.196979:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.196983:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.196987:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.196991:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.196995:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.196998:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197003:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197006:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197009:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197013:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197016:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197019:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197023:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197026:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197030:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197033:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197036:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197040:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197045:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197048:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197052:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197055:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197059:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197063:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197067:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197070:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197074:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197078:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197082:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197086:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197090:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197094:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197098:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00800000:00000080:1.0:1788572591.197101:0:166192:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00000080:00200080:1.0:1788572591.197106:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:1.0:1788572591.197113:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a4675a800 uarg=0000000000000000 00020000:00000004:1.0:1788572591.197122:0:166192:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9309, max_age 9348 00000100:00080000:1.0:1788572591.197145:0:166192:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9296 00000100:00080000:1.0:1788572591.197164:0:166192:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572591.197187:0:166192:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:1.0:1788572591.197812:0:166192:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569587456/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572591.197969:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569587456:192.168.202.157@tcp:8:lfs.0 00000100:00080000:2.0:1788572591.198189:0:166192:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a48351500 x1875444569587584/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572591.198199:0:166192:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572591.203893:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0000-osc-ffff920a47b93800: committing for last_committed 38654705666 gen 5 00000100:00080000:2.0:1788572591.203922:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0000-osc-ffff920a47b93800: connect to target with instance 9 00000100:00080000:2.0:1788572591.203935:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0000-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572591.203952:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0000_UUID 00000100:00080000:2.0:1788572591.203958:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788572591.203970:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572591.203977:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788572591.203997:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000005af9cfce): req@ffff920a48351500 x1875444569587584/t0(0) o13->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572591.207819:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569587456:192.168.202.157@tcp:8:lfs.0 00000100:00100000:3.0:1788572591.207947:0:166192:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572591.211385:0:166192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166192:1875444569587584:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572591.211425:0:166192:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:2.0:1788572591.211520:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.211545:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00020000:00000004:2.0:1788572591.211555:0:166192:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9309, max_age 9348 00000100:00080000:2.0:1788572591.211570:0:166192:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9296 00000100:00080000:2.0:1788572591.211620:0:166192:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572591.211640:0:166192:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8437760, lost: 0. 00000100:00100000:2.0:1788572591.212297:0:166192:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6680 x1875444569587712/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572591.212380:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569587712:192.168.202.157@tcp:8:lfs.0 00000100:00080000:2.0:1788572591.212523:0:166192:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff920a60ac7b80 x1875444569587840/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572591.212534:0:166192:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 0 seconds 00000100:00100000:2.0:1788572591.228408:0:2396:0:(client.c:3126:ptlrpc_free_committed()) lustre-OST0001-osc-ffff920a47b93800: committing for last_committed 0 gen 5 00000100:00080000:2.0:1788572591.228432:0:2396:0:(import.c:1176:ptlrpc_connect_interpret()) lustre-OST0001-osc-ffff920a47b93800: connect to target with instance 1 00000100:00080000:2.0:1788572591.228445:0:2396:0:(import.c:1000:ptlrpc_connect_set_flags()) lustre-OST0001-osc-ffff920a47b93800: Resetting ns_connect_flags to server flags: 0xa0427af2e3440038 00000100:00080000:2.0:1788572591.228460:0:2396:0:(import.c:1257:ptlrpc_connect_interpret()) connected to replayable target: lustre-OST0001_UUID 00000100:00080000:2.0:1788572591.228466:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to FULL 00000080:00000004:2.0:1788572591.228478:0:2396:0:(lcommon_misc.c:94:cl_ocd_update()) Changing connect_flags: 0xa0427af2e3440038 -> 0xa0427af2e3440038 00000080:00080000:2.0:1788572591.228485:0:2396:0:(lcommon_misc.c:56:cl_init_ea_size()) updating def/max_easize: 72/65536 00000100:00080000:2.0:1788572591.228507:0:2396:0:(recover.c:191:ptlrpc_wake_delayed()) @@@ waking (set 000000005af9cfce): req@ffff920a60ac7b80 x1875444569587840/t0(0) o13->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:7/4 lens 224/368 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572591.228541:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569587712:192.168.202.157@tcp:8:lfs.0 00000100:00100000:3.0:1788572591.228820:0:166192:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52896d80 going to sleep for 11 seconds 00000100:00100000:2.0:1788572591.235184:0:166192:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166192:1875444569587840:192.168.202.157@tcp:13:lfs.0 00020000:00000004:2.0:1788572591.235226:0:166192:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:2.0:1788572591.235324:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235347:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235355:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235359:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235438:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235445:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235449:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235454:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235458:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235462:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235466:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235470:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235475:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235478:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235482:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235486:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235490:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235494:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235499:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235503:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235507:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235511:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235516:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235520:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235523:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235527:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235530:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235534:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235537:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235541:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235546:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235549:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235553:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235557:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235561:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235564:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235568:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235572:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235576:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235579:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235584:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235643:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235649:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235653:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.235658:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.235661:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236445:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236461:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236469:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236473:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236477:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236482:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236486:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236490:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236495:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236498:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236502:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236505:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236509:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236512:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236515:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236518:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236522:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236525:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236528:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236531:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236535:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236538:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236542:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236546:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236550:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236553:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236557:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236560:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236564:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236568:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236572:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236575:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.236579:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.236582:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237040:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237051:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237057:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237060:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237065:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237068:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237072:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237075:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237079:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237082:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237086:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237090:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237094:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237098:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237102:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237105:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237109:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237112:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237116:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237119:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237124:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237127:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237131:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237134:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237139:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237143:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237147:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237151:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237155:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237159:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237164:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237168:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237183:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237188:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237194:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237197:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237202:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237205:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237210:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237215:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237219:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237223:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237227:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237231:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200080:2.0:1788572591.237236:0:166192:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7fffe2600340 00020000:00000080:2.0:1788572591.237241:0:166192:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a45851000 uarg=0000000000000000 00000080:00200008:2.0:1788572591.237262:0:166192:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00010000:00010000:3.0:1788572591.518395:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572592.094854:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588096:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572592.094950:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588352:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572592.094984:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [1->0] 00000100:00100000:3.0:1788572592.094991:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569587968:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572592.095020:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588224:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572592.095043:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588480:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572592.111514:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:2399:1875444569587968:192.168.202.157@tcp:400:kworker.0 00010000:00080000:2.0:1788572592.111539:0:2399:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a529b1500 x1875444569588224/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572608 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572592.111553:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588224:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572592.111566:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588480:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572592.111576:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588096:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572592.111614:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569588352:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572593.563800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572595.611724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572596.145105:0:166473:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: oleg257-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 00000100:00100000:3.0:1788572597.212053:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588736:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.212155:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588992:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.212184:0:2400:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 3 async RPCs [0->1] 00000100:00100000:3.0:1788572597.212189:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569588608:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.212214:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588864:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.212231:0:2400:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569589120:192.168.202.157@tcp:400:kworker.0 00010000:00080000:3.0:1788572597.216836:0:2400:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a529b3b80 x1875444569588864/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 0 dl 1788572613 ref 1 fl Rpc:RNQU/200/0 rc 0/0 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572597.216861:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588864:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.216877:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569589120:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.216894:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588736:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.216906:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2400:1875444569588992:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572597.216931:0:2400:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:f79285ed-4685-45a8-8731-5e2522a143f3:2400:1875444569588608:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572597.659773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00010000:0.0:1788572598.299867:0:166659:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.299873:0:166659:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7300c97bb0 00010000:00010000:0.0:1788572598.299876:0:166659:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 11 lru_type: 1 00010000:00010000:0.0:1788572598.299883:0:166659:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:0.0:1788572598.299891:0:166659:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:0.0:1788572598.299911:0:166659:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200080:0.0:1788572598.300004:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.300022:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00800000:00000004:0.0:1788572598.300031:0:166659:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9349, max_age 9355 00000100:00100000:0.0:1788572598.300083:0:166659:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589248:192.168.202.157@tcp:41:lfs.0 00000100:00100000:0.0:1788572598.300242:0:166659:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.301476:0:166659:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589248:192.168.202.157@tcp:41:lfs.0 00800000:00000004:0.0:1788572598.301503:0:166659:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: update ffff920a46419e18 cache blocks 317221/353529 objects 1023635/1024000 00000080:00200080:0.0:1788572598.301570:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.301580:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00800000:00000004:0.0:1788572598.301630:0:166659:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9349, max_age 9355 00000100:00100000:0.0:1788572598.301663:0:166659:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589376:192.168.202.157@tcp:41:lfs.0 00000100:00100000:0.0:1788572598.301720:0:166659:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00010000:00080000:0.0:1788572598.302709:0:166659:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a45b34380 x1875444569589376/t0(0) o41->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/368 e 0 to 0 dl 1788572614 ref 2 fl Rpc:RQU/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572598.302723:0:166659:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589376:192.168.202.157@tcp:41:lfs.0 00800000:00000004:0.0:1788572598.302745:0:166659:0:(obd_class.h:1116:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: update ffff920a46418458 cache blocks 317321/353529 objects 1023722/1024000 00000080:00200080:0.0:1788572598.302780:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302789:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302797:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302802:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302807:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302812:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302817:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302821:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302825:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302830:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302835:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302840:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302844:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302847:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302852:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302857:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302862:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302866:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302870:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302874:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302879:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302883:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302888:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302893:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302911:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302918:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302923:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302928:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302932:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302937:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302942:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302946:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302950:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302954:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302959:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302964:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302968:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302972:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302977:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302981:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302985:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302989:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.302994:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.302998:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303003:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303007:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303013:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303016:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303020:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303024:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303029:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303034:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303039:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303043:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303047:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303051:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303055:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303058:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303063:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303067:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303071:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303075:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303079:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303082:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303087:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303090:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303094:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303097:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303101:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303104:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303108:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303112:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303115:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303119:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303122:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303127:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303131:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303135:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303139:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303143:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303147:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303151:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303155:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303159:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303163:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303167:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303171:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303175:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303179:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303183:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303187:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303190:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303194:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303197:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303201:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303205:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303208:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303212:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303216:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303220:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303224:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303228:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303233:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303236:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303240:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303244:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303248:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303252:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303256:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303259:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303264:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303268:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303272:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303275:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303279:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303283:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303287:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303291:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303296:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303299:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303304:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303307:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303312:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303315:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303319:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00800000:00000080:0.0:1788572598.303324:0:166659:0:(lmv_obd.c:934:lmv_iocontrol()) lustre-clilmv-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.303328:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.303334:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00020000:00000004:0.0:1788572598.303343:0:166659:0:(obd_class.h:1090:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: age 9349, max_age 9355 00000100:00100000:0.0:1788572598.303367:0:166659:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589504:192.168.202.157@tcp:13:lfs.0 00000100:00100000:0.0:1788572598.303420:0:166659:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.304346:0:166659:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589504:192.168.202.157@tcp:13:lfs.0 00020000:00000004:0.0:1788572598.304361:0:166659:0:(obd_class.h:1116:obd_statfs()) lustre-OST0000-osc-ffff920a47b93800: update ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00000080:00200080:0.0:1788572598.304403:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.304412:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00020000:00000004:0.0:1788572598.304418:0:166659:0:(obd_class.h:1090:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: age 9349, max_age 9355 00000100:00100000:0.0:1788572598.304437:0:166659:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589632:192.168.202.157@tcp:13:lfs.0 00000100:00100000:0.0:1788572598.304478:0:166659:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.305371:0:166659:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166659:1875444569589632:192.168.202.157@tcp:13:lfs.0 00020000:00000004:0.0:1788572598.305380:0:166659:0:(obd_class.h:1116:obd_statfs()) lustre-OST0001-osc-ffff920a47b93800: update ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00200080:0.0:1788572598.305408:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305415:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305419:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305422:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305425:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305427:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305430:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305433:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305436:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305440:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305443:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305445:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305447:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305449:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305451:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305453:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305456:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305458:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305460:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305463:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305465:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305467:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305469:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305471:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305473:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305476:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305478:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305480:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305482:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305484:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305486:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305488:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305490:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305492:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305497:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305501:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305505:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305509:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305514:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305518:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305523:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305527:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305547:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305552:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305556:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305560:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305564:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305568:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305572:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305575:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305580:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305583:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305613:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305619:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305624:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305627:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305631:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305635:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305638:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305641:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305644:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305646:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305649:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305651:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305653:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305655:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305657:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305658:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305661:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305663:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305665:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305668:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305670:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305678:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305682:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305684:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305686:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305687:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305689:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305691:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305693:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305695:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305697:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305699:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305703:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305705:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305707:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305710:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305712:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305714:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305716:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305718:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305720:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305722:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305725:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305726:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305728:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305730:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305732:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305734:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305736:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305737:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305739:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305741:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305743:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305744:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305746:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305748:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305750:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305752:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305754:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305756:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305758:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305759:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305761:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305763:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305765:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305766:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305768:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305770:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305772:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305773:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305775:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305777:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200080:0.0:1788572598.305779:0:166659:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c00866a4 arg=7ffdc911a560 00020000:00000080:0.0:1788572598.305781:0:166659:0:(lov_obd.c:976:lov_iocontrol()) lustre-clilov-ffff920a47b93800: cmd=c00866a4 len=592 karg=ffff920a47900c00 uarg=0000000000000000 00000080:00200008:0.0:1788572598.305797:0:166659:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 100000 00000080:00200000:2.0:1788572598.318199:0:166675:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:2.0:1788572598.318238:0:166675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572598.318263:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572598.318283:0:166675:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00000080:00200000:2.0:1788572598.318298:0:166675:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572598.318307:0:166675:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:2.0:1788572598.318311:0:166675:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:2.0:1788572598.318320:0:166675:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf730180bbf8 00010000:00010000:2.0:1788572598.318324:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 12 lru_type: 1 00010000:00010000:2.0:1788572598.318335:0:166675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:2.0:1788572598.318352:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.318366:0:166675:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=27 00000080:00200000:2.0:1788572598.318371:0:166675:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:2.0:1788572598.318387:0:166675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:2.0:1788572598.318442:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.318449:0:166675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.318458:0:166675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572598.318482:0:166675:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569589760:192.168.202.157@tcp:101:lfs.0 00000100:00100000:2.0:1788572598.318602:0:166675:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a526450c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.321446:0:166675:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7b80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569589760:192.168.202.157@tcp:101:lfs.0 00010000:00010000:0.0:1788572598.321477:0:166675:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321499:0:166675:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321508:0:166675:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321523:0:166675:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321537:0:166675:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321548:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321554:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321558:0:166675:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.321567:0:166675:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a60ac7b80 x1875444569589760/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.321578:0:166675:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321600:0:166675:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/7 (0 0) 00000080:00200000:0.0:1788572598.321606:0:166675:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.321642:0:166675:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00200000:0.0:1788572598.321647:0:166675:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x7:0x0](ffff920a50d49988): rc = 0 00000080:00010000:0.0:1788572598.321655:0:166675:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb66f 00000080:00010000:0.0:1788572598.321666:0:166675:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:0.0:1788572598.321670:0:166675:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb66f from it ffffaf730180bac8 00010000:00010000:0.0:1788572598.321672:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321680:0:166675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.321714:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.321732:0:166675:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), file ffff920a42ca9500, open_flags 18800, mode 0 opened 0 00000080:00200000:0.0:1788572598.321738:0:166675:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), intent=open 00000002:00010000:0.0:1788572598.321758:0:166675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x7:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572598.321816:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.321825:0:166675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.321836:0:166675:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.321844:0:166675:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34380 x1875444569590016/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 656/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.321853:0:166675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.321872:0:166675:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590016:192.168.202.157@tcp:101:lfs.0 00000100:00100000:0.0:1788572598.322094:0:166675:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.326143:0:166675:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590016:192.168.202.157@tcp:101:lfs.0 00010000:00010000:0.0:1788572598.326169:0:166675:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326196:0:166675:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326204:0:166675:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326216:0:166675:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326226:0:166675:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326239:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326248:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326256:0:166675:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.326268:0:166675:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0xb, status=0 req@ffff920a45b34380 x1875444569590016/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 656/608 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.326281:0:166675:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326290:0:166675:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 9663686441/43 (0 0) 00000002:00010000:0.0:1788572598.326305:0:166675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x240002729:0x2b:0x0], intent: open flags 0304001 00010000:00010000:0.0:1788572598.326344:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.326351:0:166675:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.326361:0:166675:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.326367:0:166675:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b37800 x1875444569590272/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.326375:0:166675:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.326402:0:166675:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590272:192.168.202.157@tcp:101:lfs.0 00000100:00100000:0.0:1788572598.326560:0:166675:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:3.0:1788572598.335289:0:166675:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37800 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590272:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572598.335308:0:166675:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335321:0:166675:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335329:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335337:0:166675:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 3/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335350:0:166675:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335357:0:166675:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a51a70a00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572598.335362:0:166675:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335369:0:166675:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 4/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335376:0:166675:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 5/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335390:0:166675:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572598.335398:0:166675:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 3/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335404:0:166675:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb67d lrc: 0/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 166675 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572598.335430:0:166675:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a45b37800 x1875444569590272/t30064771073(30064771073) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572598.335439:0:166675:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37800 x1875444569590272/t30064771073(30064771073) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572598.335500:0:166675:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x7:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572598.335522:0:166675:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x2b:0x0], updating i_size 4096 00000080:00200000:3.0:1788572598.335525:0:166675:0:(namei.c:147:ll_iget()) got inode: [0x240002729:0x2b:0x0](ffff920a50d44b08): rc = 0 00000080:00010000:3.0:1788572598.335530:0:166675:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb676 00000080:00010000:3.0:1788572598.335538:0:166675:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00200008:3.0:1788572598.335548:0:166675:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 304000 00000002:00100000:3.0:1788572598.335565:0:166675:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37800 x1875444569590272/t30064771073(30064771073) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572598.335630:0:166675:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb676 from it ffff920a52f77e40 00010000:00010000:3.0:1788572598.335636:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.335645:0:166675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572598.335663:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 44 00000080:00200000:3.0:1788572598.335702:0:166675:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572598.335717:0:166675:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572598.335727:0:166675:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572598.335736:0:166675:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00000080:00200000:3.0:1788572598.335742:0:166675:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572598.335746:0:166675:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572598.335749:0:166675:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:3.0:1788572598.335752:0:166675:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf730180bbf8 00010000:00010000:3.0:1788572598.335754:0:166675:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 13 lru_type: 1 00010000:00010000:3.0:1788572598.335759:0:166675:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:3.0:1788572598.335769:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572598.335773:0:166675:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=24 00000080:00200000:3.0:1788572598.335777:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572598.335780:0:166675:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=264 00000080:00200000:3.0:1788572598.335783:0:166675:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572598.335786:0:166675:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 504000 00000080:00200008:3.0:1788572598.335799:0:166675:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 304000 00000080:00200080:3.0:1788572598.335820:0:166675:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08) cmd=800466af arg=7ffdc90d34ac 00000080:00200008:3.0:1788572598.335881:0:166675:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 504000 00000002:00100000:3.0:1788572598.335928:0:166675:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b37800 x1875444569590272/t30064771073(30064771073) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572598.335939:0:166675:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572598.335945:0:166675:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b1180 x1875444569590528/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572598.335965:0:166675:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590528:192.168.202.157@tcp:35:lfs.0 00000100:00100000:3.0:1788572598.336034:0:166675:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ada80 going to sleep for 11 seconds 00010000:00080000:0.0:1788572598.337002:0:166675:0:(ldlm_request.c:1671:ldlm_cli_update_pool()) @@@ Zero SLV or limit found (SLV=0, limit=46162) req@ffff920a529b1180 x1875444569590528/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/456 e 0 to 0 dl 1788572614 ref 2 fl Rpc:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572598.337016:0:166675:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166675:1875444569590528:192.168.202.157@tcp:35:lfs.0 00000002:00100000:0.0:1788572598.337037:0:166675:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b37800 x1875444569590272/t30064771073(30064771073) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.343639:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.343668:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.343688:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788572598.343703:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00000080:00200000:0.0:1788572598.343716:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.343724:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.343728:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.343735:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.343739:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 14 lru_type: 1 00010000:00010000:0.0:1788572598.343748:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788572598.343760:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.343767:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.343772:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.343775:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.343779:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.343789:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.343794:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-0, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a460acf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.343802:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-0, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.343821:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-0,[0x240002729:0x2c:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.343870:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.343879:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.343890:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.343913:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b37800 x1875444569590784/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.343924:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.343944:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569590784:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.344055:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.347394:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569590784:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.347420:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347446:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347456:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347468:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347479:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347491:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347500:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347514:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.347527:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b37800 x1875444569590784/t30064771075(30064771075) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.347540:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37800 x1875444569590784/t30064771075(30064771075) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.347554:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.347566:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347576:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/44 (0 0) 00000080:00200000:0.0:1788572598.347582:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.347634:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-0, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.347658:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x2c:0x0](ffff920a50d463c8) 00000080:00200000:0.0:1788572598.347730:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.347735:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x2c:0x0](ffff920a50d463c8) 00000080:00010000:0.0:1788572598.347739:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2c:0x0](ffff920a50d463c8) for lock 0x8364c713e25bb684 00000080:00010000:0.0:1788572598.347748:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2c:0x0](ffff920a50d463c8) 00000080:00200008:0.0:1788572598.347759:0:166681:0:(file.c:1090:ll_file_open()) START file f-0:[0x240002729:0x2c:0x0](ffff920a50d463c8), flags 100102 00000002:00100000:0.0:1788572598.347779:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37800 x1875444569590784/t30064771075(30064771075) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.347808:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb684 from it ffff920a52f77c60 00010000:00010000:0.0:1788572598.347813:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.347821:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.347839:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2c:0x0](ffff920a50d463c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.347859:0:166681:0:(file.c:479:ll_file_release()) START file f-0:[0x240002729:0x2c:0x0](ffff920a50d463c8), flags 100002 00000002:00100000:0.0:1788572598.347895:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b37800 x1875444569590784/t30064771075(30064771075) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.347918:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.347923:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b37480 x1875444569591040/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.347941:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591040:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.348005:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.349193:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37480 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591040:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.349208:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b37800 x1875444569590784/t30064771075(30064771075) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.349249:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.349264:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.349276:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572598.349288:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00000080:00200000:0.0:1788572598.349301:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.349307:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.349311:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.349321:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.349325:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 15 lru_type: 1 00010000:00010000:0.0:1788572598.349334:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:0.0:1788572598.349345:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.349350:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.349355:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.349358:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.349360:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.349368:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.349373:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-1, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a460ac180, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.349380:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-1, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.349397:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-1,[0x240002729:0x2d:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.349435:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.349443:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.349453:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.349459:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34000 x1875444569591296/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.349468:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.349481:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591296:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.349567:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6cc0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.351317:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591296:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.351339:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351364:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351378:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351389:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351400:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351412:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351421:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351435:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.351447:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b34000 x1875444569591296/t30064771077(30064771077) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.351459:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34000 x1875444569591296/t30064771077(30064771077) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.351471:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.351483:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351493:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/45 (0 0) 00000080:00200000:0.0:1788572598.351499:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.351508:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-1, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.351532:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x2d:0x0](ffff920a50d43a88) 00000080:00200000:0.0:1788572598.351637:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.351643:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x2d:0x0](ffff920a50d43a88) 00000080:00010000:0.0:1788572598.351646:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2d:0x0](ffff920a50d43a88) for lock 0x8364c713e25bb68b 00000080:00010000:0.0:1788572598.351655:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2d:0x0](ffff920a50d43a88) 00000080:00200008:0.0:1788572598.351665:0:166681:0:(file.c:1090:ll_file_open()) START file f-1:[0x240002729:0x2d:0x0](ffff920a50d43a88), flags 100102 00000002:00100000:0.0:1788572598.351682:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34000 x1875444569591296/t30064771077(30064771077) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.351712:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb68b from it ffff920a52f77c60 00010000:00010000:0.0:1788572598.351717:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.351727:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.351753:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2d:0x0](ffff920a50d43a88), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.351768:0:166681:0:(file.c:479:ll_file_release()) START file f-1:[0x240002729:0x2d:0x0](ffff920a50d43a88), flags 100002 00000002:00100000:0.0:1788572598.351806:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b34000 x1875444569591296/t30064771077(30064771077) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.351815:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.351819:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b35c00 x1875444569591552/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.351835:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35c00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591552:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.351886:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6e40 going to sleep for 11 seconds 00000100:00100000:3.0:1788572598.352951:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35c00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591552:192.168.202.157@tcp:35:createmany.0 00000002:00100000:3.0:1788572598.352974:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b34000 x1875444569591296/t30064771077(30064771077) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572598.353022:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572598.353036:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572598.353049:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572598.353066:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00000080:00200000:3.0:1788572598.353074:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572598.353080:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572598.353083:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:3.0:1788572598.353089:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:3.0:1788572598.353092:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 16 lru_type: 1 00010000:00010000:3.0:1788572598.353101:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:3.0:1788572598.353113:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572598.353119:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:3.0:1788572598.353126:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:3.0:1788572598.353130:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:3.0:1788572598.353133:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572598.353143:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:3.0:1788572598.353148:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-2, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a51dc1200, open_flags 8042, mode 81a4 opened 0 00000080:00200000:3.0:1788572598.353155:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-2, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:3.0:1788572598.353172:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-2,[0x240002729:0x2e:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:3.0:1788572598.353207:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.353215:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:3.0:1788572598.353225:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:3.0:1788572598.353231:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b3800 x1875444569591808/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:3.0:1788572598.353241:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572598.353256:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591808:192.168.202.157@tcp:101:createmany.0 00000100:00100000:3.0:1788572598.353341:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ad300 going to sleep for 11 seconds 00000100:00100000:3.0:1788572598.355004:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569591808:192.168.202.157@tcp:101:createmany.0 00010000:00010000:3.0:1788572598.355032:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355066:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355075:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355086:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355097:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355110:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355124:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355129:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572598.355137:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a529b3800 x1875444569591808/t30064771079(30064771079) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:3.0:1788572598.355145:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a529b3800 x1875444569591808/t30064771079(30064771079) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572598.355155:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:3.0:1788572598.355164:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355171:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/46 (0 0) 00000080:00200000:3.0:1788572598.355175:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:3.0:1788572598.355183:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-2, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:3.0:1788572598.355201:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x2e:0x0](ffff920a50d3a1c8) 00000080:00200000:3.0:1788572598.355266:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:3.0:1788572598.355271:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x2e:0x0](ffff920a50d3a1c8) 00000080:00010000:3.0:1788572598.355274:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2e:0x0](ffff920a50d3a1c8) for lock 0x8364c713e25bb692 00000080:00010000:3.0:1788572598.355281:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2e:0x0](ffff920a50d3a1c8) 00000080:00200008:3.0:1788572598.355292:0:166681:0:(file.c:1090:ll_file_open()) START file f-2:[0x240002729:0x2e:0x0](ffff920a50d3a1c8), flags 100102 00000002:00100000:3.0:1788572598.355311:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a529b3800 x1875444569591808/t30064771079(30064771079) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:3.0:1788572598.355346:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb692 from it ffff920a51b4e780 00010000:00010000:3.0:1788572598.355351:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.355360:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572598.355378:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2e:0x0](ffff920a50d3a1c8), inode mode 81a4 mask 40 00000080:00200008:3.0:1788572598.355398:0:166681:0:(file.c:479:ll_file_release()) START file f-2:[0x240002729:0x2e:0x0](ffff920a50d3a1c8), flags 100002 00000002:00100000:3.0:1788572598.355438:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a529b3800 x1875444569591808/t30064771079(30064771079) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:3.0:1788572598.355446:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:3.0:1788572598.355450:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b1880 x1875444569592064/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:3.0:1788572598.355468:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592064:192.168.202.157@tcp:35:createmany.0 00000100:00100000:3.0:1788572598.355518:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a585ada80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.356561:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592064:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.356577:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a529b3800 x1875444569591808/t30064771079(30064771079) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.356665:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.356678:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.356691:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788572598.356703:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00000080:00200000:0.0:1788572598.356722:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.356728:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.356732:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.356737:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.356741:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 17 lru_type: 1 00010000:00010000:0.0:1788572598.356750:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788572598.356760:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.356765:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.356770:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.356773:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.356775:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.356782:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.356785:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-3, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a53456300, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.356791:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-3, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.356804:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-3,[0x240002729:0x2f:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.356836:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.356842:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.356848:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.356852:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b36300 x1875444569592320/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.356859:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.356869:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592320:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.356958:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.360281:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592320:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.360309:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360331:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360337:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360345:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360352:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360362:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360369:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360381:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.360392:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b36300 x1875444569592320/t30064771081(30064771081) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.360405:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b36300 x1875444569592320/t30064771081(30064771081) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.360418:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.360427:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360434:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/47 (0 0) 00000080:00200000:0.0:1788572598.360439:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.360449:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-3, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.360468:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x2f:0x0](ffff920a50d41148) 00000080:00200000:0.0:1788572598.360531:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.360534:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x2f:0x0](ffff920a50d41148) 00000080:00010000:0.0:1788572598.360537:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2f:0x0](ffff920a50d41148) for lock 0x8364c713e25bb699 00000080:00010000:0.0:1788572598.360543:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2f:0x0](ffff920a50d41148) 00000080:00200008:0.0:1788572598.360552:0:166681:0:(file.c:1090:ll_file_open()) START file f-3:[0x240002729:0x2f:0x0](ffff920a50d41148), flags 100102 00000002:00100000:0.0:1788572598.360566:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b36300 x1875444569592320/t30064771081(30064771081) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.360622:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb699 from it ffff920a52f77c60 00010000:00010000:0.0:1788572598.360626:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.360632:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.360646:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2f:0x0](ffff920a50d41148), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.360661:0:166681:0:(file.c:479:ll_file_release()) START file f-3:[0x240002729:0x2f:0x0](ffff920a50d41148), flags 100002 00000002:00100000:0.0:1788572598.360693:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b36300 x1875444569592320/t30064771081(30064771081) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.360703:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.360710:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b37100 x1875444569592576/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.360728:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592576:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.360788:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba60c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.362010:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592576:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.362033:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b36300 x1875444569592320/t30064771081(30064771081) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.362085:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.362099:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.362111:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788572598.362123:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00000080:00200000:0.0:1788572598.362137:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.362143:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.362146:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.362151:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.362155:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 18 lru_type: 1 00010000:00010000:0.0:1788572598.362164:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:0.0:1788572598.362176:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.362182:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.362189:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.362194:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.362200:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.362210:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.362215:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-4, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a53456000, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.362222:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-4, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.362239:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-4,[0x240002729:0x30:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.362280:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.362289:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.362298:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.362306:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34a80 x1875444569592832/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.362316:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.362331:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34a80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592832:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.362419:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.365409:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34a80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569592832:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572598.365430:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365451:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x30:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365458:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365467:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365475:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365485:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365491:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365496:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572598.365504:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b34a80 x1875444569592832/t30064771083(30064771083) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572598.365518:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34a80 x1875444569592832/t30064771083(30064771083) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572598.365527:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572598.365536:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365543:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/48 (0 0) 00000080:00200000:1.0:1788572598.365548:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572598.365557:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-4, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572598.365577:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x30:0x0](ffff920a50d4a1c8) 00000080:00200000:1.0:1788572598.365668:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x30:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572598.365673:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x30:0x0](ffff920a50d4a1c8) 00000080:00010000:1.0:1788572598.365676:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x30:0x0](ffff920a50d4a1c8) for lock 0x8364c713e25bb6a0 00000080:00010000:1.0:1788572598.365683:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x30:0x0](ffff920a50d4a1c8) 00000080:00200008:1.0:1788572598.365692:0:166681:0:(file.c:1090:ll_file_open()) START file f-4:[0x240002729:0x30:0x0](ffff920a50d4a1c8), flags 100102 00000002:00100000:1.0:1788572598.365708:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34a80 x1875444569592832/t30064771083(30064771083) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572598.365736:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6a0 from it ffff920a52f77e40 00010000:00010000:1.0:1788572598.365741:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.365750:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572598.365773:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x30:0x0](ffff920a50d4a1c8), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572598.365790:0:166681:0:(file.c:479:ll_file_release()) START file f-4:[0x240002729:0x30:0x0](ffff920a50d4a1c8), flags 100002 00000002:00100000:1.0:1788572598.365830:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b34a80 x1875444569592832/t30064771083(30064771083) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572598.365840:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572598.365846:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352a00 x1875444569593088/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572598.365865:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593088:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572598.365937:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cb180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.368108:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593088:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572598.368134:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b34a80 x1875444569592832/t30064771083(30064771083) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572598.368177:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572598.368190:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.368204:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572598.368214:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00000080:00200000:1.0:1788572598.368220:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.368224:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572598.368227:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572598.368233:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:1.0:1788572598.368237:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 19 lru_type: 1 00010000:00010000:1.0:1788572598.368258:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572598.368271:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.368278:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:1.0:1788572598.368283:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.368287:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572598.368290:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.368299:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572598.368303:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-5, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a46872600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572598.368311:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-5, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572598.368329:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-5,[0x240002729:0x31:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572598.368376:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.368385:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572598.368395:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572598.368402:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350e00 x1875444569593344/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572598.368412:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572598.368436:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593344:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572598.368531:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cb180 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.375562:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593344:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572598.375582:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375631:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x31:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375641:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375650:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375658:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375668:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375673:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375678:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572598.375685:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48350e00 x1875444569593344/t30064771085(30064771085) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572598.375694:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350e00 x1875444569593344/t30064771085(30064771085) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572598.375703:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572598.375711:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375719:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/49 (0 0) 00000080:00200000:1.0:1788572598.375723:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572598.375733:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-5, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572598.375756:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x31:0x0](ffff920a50d49148) 00000080:00200000:1.0:1788572598.375823:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x31:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572598.375828:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x31:0x0](ffff920a50d49148) 00000080:00010000:1.0:1788572598.375838:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x31:0x0](ffff920a50d49148) for lock 0x8364c713e25bb6a7 00000080:00010000:1.0:1788572598.375848:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x31:0x0](ffff920a50d49148) 00000080:00200008:1.0:1788572598.375859:0:166681:0:(file.c:1090:ll_file_open()) START file f-5:[0x240002729:0x31:0x0](ffff920a50d49148), flags 100102 00000002:00100000:1.0:1788572598.375878:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350e00 x1875444569593344/t30064771085(30064771085) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572598.375918:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6a7 from it ffff920a48138b40 00010000:00010000:1.0:1788572598.375923:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.375932:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572598.375959:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x31:0x0](ffff920a50d49148), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572598.375983:0:166681:0:(file.c:479:ll_file_release()) START file f-5:[0x240002729:0x31:0x0](ffff920a50d49148), flags 100002 00000002:00100000:1.0:1788572598.376023:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48350e00 x1875444569593344/t30064771085(30064771085) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572598.376032:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572598.376036:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352680 x1875444569593600/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572598.376055:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593600:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572598.376122:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cba80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.381670:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593600:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572598.381712:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48350e00 x1875444569593344/t30064771085(30064771085) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572598.381784:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572598.381804:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.381822:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572598.381837:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00000080:00200000:1.0:1788572598.381846:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.381853:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572598.381857:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572598.381864:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:1.0:1788572598.381870:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 20 lru_type: 1 00010000:00010000:1.0:1788572598.381880:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572598.381910:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.381922:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:1.0:1788572598.381928:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.381933:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572598.381938:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.381950:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572598.381955:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-6, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a46872c00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572598.381963:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-6, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572598.381981:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-6,[0x240002729:0x32:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572598.382027:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.382035:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572598.382045:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572598.382051:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48350000 x1875444569593856/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572598.382061:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572598.382079:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593856:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572598.382183:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cba80 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.388255:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350000 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569593856:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572598.388281:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388308:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/0,1 mode: --/PR res: [0x240002729:0x32:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388317:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/0,1 mode: --/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388334:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388345:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388358:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388383:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388391:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572598.388412:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48350000 x1875444569593856/t30064771087(30064771087) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572598.388426:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569593856/t30064771087(30064771087) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572598.388440:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572598.388453:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388464:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/50 (0 0) 00000080:00200000:1.0:1788572598.388470:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572598.388482:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-6, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572598.388509:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x32:0x0](ffff920a50d4ba88) 00000080:00200000:1.0:1788572598.388662:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x32:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572598.388667:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x32:0x0](ffff920a50d4ba88) 00000080:00010000:1.0:1788572598.388672:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x32:0x0](ffff920a50d4ba88) for lock 0x8364c713e25bb6ae 00000080:00010000:1.0:1788572598.388681:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x32:0x0](ffff920a50d4ba88) 00000080:00200008:1.0:1788572598.388695:0:166681:0:(file.c:1090:ll_file_open()) START file f-6:[0x240002729:0x32:0x0](ffff920a50d4ba88), flags 100102 00000002:00100000:1.0:1788572598.388714:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48350000 x1875444569593856/t30064771087(30064771087) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572598.388760:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6ae from it ffff920a48138b40 00010000:00010000:1.0:1788572598.388765:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.388775:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572598.388801:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x32:0x0](ffff920a50d4ba88), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572598.388827:0:166681:0:(file.c:479:ll_file_release()) START file f-6:[0x240002729:0x32:0x0](ffff920a50d4ba88), flags 100002 00000002:00100000:1.0:1788572598.388869:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48350000 x1875444569593856/t30064771087(30064771087) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572598.388880:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572598.388885:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352d80 x1875444569594112/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572598.388920:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594112:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572598.388991:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cbb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.390258:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352d80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594112:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572598.390280:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48350000 x1875444569593856/t30064771087(30064771087) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572598.390346:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572598.390361:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.390527:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572598.390540:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00000080:00200000:1.0:1788572598.390548:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.390554:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572598.390557:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572598.390563:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:1.0:1788572598.390566:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 21 lru_type: 1 00010000:00010000:1.0:1788572598.390574:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572598.390610:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.390618:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:1.0:1788572598.390626:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.390631:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572598.390635:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.390647:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572598.390652:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-7, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a46873e00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572598.390659:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-7, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572598.390679:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-7,[0x240002729:0x33:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572598.390722:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.390730:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572598.390740:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572598.390746:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351880 x1875444569594368/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572598.390754:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572598.390769:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594368:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572598.390872:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cbb40 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.392761:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594368:192.168.202.157@tcp:101:createmany.0 00010000:00010000:1.0:1788572598.392783:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392810:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x33:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392819:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392830:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392839:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392853:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392861:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392868:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572598.392878:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48351880 x1875444569594368/t30064771089(30064771089) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572598.392891:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351880 x1875444569594368/t30064771089(30064771089) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572598.392918:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1788572598.392930:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.392940:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/51 (0 0) 00000080:00200000:1.0:1788572598.392946:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572598.392957:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-7, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:1.0:1788572598.392980:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x33:0x0](ffff920a50d48908) 00000080:00200000:1.0:1788572598.393059:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x33:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:1.0:1788572598.393064:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x33:0x0](ffff920a50d48908) 00000080:00010000:1.0:1788572598.393067:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x33:0x0](ffff920a50d48908) for lock 0x8364c713e25bb6b5 00000080:00010000:1.0:1788572598.393076:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x33:0x0](ffff920a50d48908) 00000080:00200008:1.0:1788572598.393087:0:166681:0:(file.c:1090:ll_file_open()) START file f-7:[0x240002729:0x33:0x0](ffff920a50d48908), flags 100102 00000002:00100000:1.0:1788572598.393104:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351880 x1875444569594368/t30064771089(30064771089) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:1.0:1788572598.393137:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6b5 from it ffff920a48138b40 00010000:00010000:1.0:1788572598.393141:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.393151:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572598.393175:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x33:0x0](ffff920a50d48908), inode mode 81a4 mask 40 00000080:00200008:1.0:1788572598.393200:0:166681:0:(file.c:479:ll_file_release()) START file f-7:[0x240002729:0x33:0x0](ffff920a50d48908), flags 100002 00000002:00100000:1.0:1788572598.393236:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48351880 x1875444569594368/t30064771089(30064771089) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1788572598.393245:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1788572598.393250:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351180 x1875444569594624/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:1.0:1788572598.393267:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594624:192.168.202.157@tcp:35:createmany.0 00000100:00100000:1.0:1788572598.393321:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cb0c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.394718:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594624:192.168.202.157@tcp:35:createmany.0 00000002:00100000:1.0:1788572598.394742:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48351880 x1875444569594368/t30064771089(30064771089) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572598.394805:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572598.394820:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572598.394834:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572598.394846:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00000080:00200000:1.0:1788572598.394854:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572598.394859:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572598.394863:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:1.0:1788572598.394869:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:1.0:1788572598.394873:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 22 lru_type: 1 00010000:00010000:1.0:1788572598.394881:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:1.0:1788572598.394910:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.394917:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:1.0:1788572598.394922:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.394926:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:1.0:1788572598.394929:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572598.394940:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:1.0:1788572598.394945:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-8, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a46872600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:1.0:1788572598.394952:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-8, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:1.0:1788572598.394970:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-8,[0x240002729:0x34:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:1.0:1788572598.395015:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.395023:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572598.395032:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572598.395039:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48352300 x1875444569594880/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572598.395048:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572598.395063:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594880:192.168.202.157@tcp:101:createmany.0 00000100:00100000:1.0:1788572598.395170:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a458cb0c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.397001:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48352300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569594880:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.397030:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397058:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/0,1 mode: --/PR res: [0x240002729:0x34:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397067:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/0,1 mode: --/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397079:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397097:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397109:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397117:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397130:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.397141:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a48352300 x1875444569594880/t30064771091(30064771091) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.397155:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48352300 x1875444569594880/t30064771091(30064771091) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.397168:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.397185:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397195:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/52 (0 0) 00000080:00200000:0.0:1788572598.397201:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.397212:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-8, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.397238:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x34:0x0](ffff920a50d40908) 00000080:00200000:0.0:1788572598.397300:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x34:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.397304:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x34:0x0](ffff920a50d40908) 00000080:00010000:0.0:1788572598.397308:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x34:0x0](ffff920a50d40908) for lock 0x8364c713e25bb6bc 00000080:00010000:0.0:1788572598.397316:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x34:0x0](ffff920a50d40908) 00000080:00200008:0.0:1788572598.397328:0:166681:0:(file.c:1090:ll_file_open()) START file f-8:[0x240002729:0x34:0x0](ffff920a50d40908), flags 100102 00000002:00100000:0.0:1788572598.397353:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48352300 x1875444569594880/t30064771091(30064771091) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.397384:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6bc from it ffff920a48138b40 00010000:00010000:0.0:1788572598.397390:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.397400:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.397420:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x34:0x0](ffff920a50d40908), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.397438:0:166681:0:(file.c:479:ll_file_release()) START file f-8:[0x240002729:0x34:0x0](ffff920a50d40908), flags 100002 00000002:00100000:0.0:1788572598.397490:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a48352300 x1875444569594880/t30064771091(30064771091) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.397499:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.397504:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b36680 x1875444569595136/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.397536:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595136:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.397688:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.398725:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595136:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.398743:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a48352300 x1875444569594880/t30064771091(30064771091) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.398785:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.398796:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.398806:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788572598.398815:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00000080:00200000:0.0:1788572598.398824:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.398828:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.398830:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.398834:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.398836:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 23 lru_type: 1 00010000:00010000:0.0:1788572598.398841:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788572598.398849:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.398853:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.398857:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.398860:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.398863:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.398873:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.398877:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-9, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845cf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.398883:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-9, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.398910:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-9,[0x240002729:0x35:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.398950:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.398958:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.398967:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.398973:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b37b80 x1875444569595392/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.398982:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.398996:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b37b80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595392:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.399108:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5f00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.400865:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b37b80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595392:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.400886:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400925:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x35:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400934:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400945:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400956:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400968:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400977:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.400989:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.401000:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b37b80 x1875444569595392/t30064771093(30064771093) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.401012:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37b80 x1875444569595392/t30064771093(30064771093) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.401025:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.401047:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.401057:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/53 (0 0) 00000080:00200000:0.0:1788572598.401070:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.401093:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-9, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.401120:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x35:0x0](ffff920a50d47448) 00000080:00200000:0.0:1788572598.401180:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x35:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.401184:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x35:0x0](ffff920a50d47448) 00000080:00010000:0.0:1788572598.401188:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x35:0x0](ffff920a50d47448) for lock 0x8364c713e25bb6c3 00000080:00010000:0.0:1788572598.401195:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x35:0x0](ffff920a50d47448) 00000080:00200008:0.0:1788572598.401207:0:166681:0:(file.c:1090:ll_file_open()) START file f-9:[0x240002729:0x35:0x0](ffff920a50d47448), flags 100102 00000002:00100000:0.0:1788572598.401224:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b37b80 x1875444569595392/t30064771093(30064771093) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.401261:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6c3 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.401266:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.401278:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.401296:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x35:0x0](ffff920a50d47448), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.401320:0:166681:0:(file.c:479:ll_file_release()) START file f-9:[0x240002729:0x35:0x0](ffff920a50d47448), flags 100002 00000002:00100000:0.0:1788572598.401355:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b37b80 x1875444569595392/t30064771093(30064771093) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.401364:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.401369:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34700 x1875444569595648/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.401399:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595648:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.401461:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.405029:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595648:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.405054:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b37b80 x1875444569595392/t30064771093(30064771093) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.405119:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.405132:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.405146:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788572598.405157:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00000080:00200000:0.0:1788572598.405167:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.405172:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.405175:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.405182:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.405186:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 24 lru_type: 1 00010000:00010000:0.0:1788572598.405195:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788572598.405209:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.405215:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.405220:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.405224:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.405227:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.405238:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.405243:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-10, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845c600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.405251:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-10, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.405269:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-10,[0x240002729:0x36:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.405313:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.405320:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.405327:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.405331:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b35500 x1875444569595904/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.405338:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.405366:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595904:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.405515:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5c00 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.407267:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35500 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569595904:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.407288:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407315:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/0,1 mode: --/PR res: [0x240002729:0x36:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407324:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/0,1 mode: --/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407335:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407345:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407357:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407366:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407378:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.407389:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b35500 x1875444569595904/t30064771095(30064771095) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.407402:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b35500 x1875444569595904/t30064771095(30064771095) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.407414:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.407426:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407436:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/54 (0 0) 00000080:00200000:0.0:1788572598.407448:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.407459:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-10, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.407481:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x36:0x0](ffff920a50d41988) 00000080:00200000:0.0:1788572598.407548:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x36:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.407553:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x36:0x0](ffff920a50d41988) 00000080:00010000:0.0:1788572598.407556:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x36:0x0](ffff920a50d41988) for lock 0x8364c713e25bb6ca 00000080:00010000:0.0:1788572598.407565:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x36:0x0](ffff920a50d41988) 00000080:00200008:0.0:1788572598.407577:0:166681:0:(file.c:1090:ll_file_open()) START file f-10:[0x240002729:0x36:0x0](ffff920a50d41988), flags 100102 00000002:00100000:0.0:1788572598.407628:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b35500 x1875444569595904/t30064771095(30064771095) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.407661:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6ca from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.407666:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.407675:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.407694:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x36:0x0](ffff920a50d41988), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.407721:0:166681:0:(file.c:479:ll_file_release()) START file f-10:[0x240002729:0x36:0x0](ffff920a50d41988), flags 100002 00000002:00100000:0.0:1788572598.407768:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b35500 x1875444569595904/t30064771095(30064771095) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.407777:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.407782:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b36a00 x1875444569596160/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.407802:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596160:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.407858:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.412208:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36a00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596160:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.412252:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b35500 x1875444569595904/t30064771095(30064771095) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.412336:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.412354:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.412371:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788572598.412385:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00000080:00200000:0.0:1788572598.412406:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.412412:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.412416:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.412423:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.412427:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 25 lru_type: 1 00010000:00010000:0.0:1788572598.412435:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788572598.412457:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.412464:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.412470:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.412473:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.412477:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.412491:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.412495:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-11, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845dc80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.412502:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-11, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.412520:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-11,[0x240002729:0x37:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.412568:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.412577:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.412625:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.412633:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b35180 x1875444569596416/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.412643:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.412663:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596416:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.412790:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5d80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.417157:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596416:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.417185:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417215:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x37:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417225:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417236:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417248:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417261:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417270:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417283:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.417294:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b35180 x1875444569596416/t30064771097(30064771097) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.417306:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b35180 x1875444569596416/t30064771097(30064771097) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.417317:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.417331:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417341:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/55 (0 0) 00000080:00200000:0.0:1788572598.417347:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.417359:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-11, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.417390:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x37:0x0](ffff920a50d42a08) 00000080:00200000:0.0:1788572598.417446:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x37:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.417451:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x37:0x0](ffff920a50d42a08) 00000080:00010000:0.0:1788572598.417454:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x37:0x0](ffff920a50d42a08) for lock 0x8364c713e25bb6d1 00000080:00010000:0.0:1788572598.417463:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x37:0x0](ffff920a50d42a08) 00000080:00200008:0.0:1788572598.417475:0:166681:0:(file.c:1090:ll_file_open()) START file f-11:[0x240002729:0x37:0x0](ffff920a50d42a08), flags 100102 00000002:00100000:0.0:1788572598.417494:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b35180 x1875444569596416/t30064771097(30064771097) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.417527:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6d1 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.417531:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.417658:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.417674:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x37:0x0](ffff920a50d42a08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.417696:0:166681:0:(file.c:479:ll_file_release()) START file f-11:[0x240002729:0x37:0x0](ffff920a50d42a08), flags 100002 00000002:00100000:0.0:1788572598.417728:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b35180 x1875444569596416/t30064771097(30064771097) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.417735:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.417739:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b35880 x1875444569596672/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.417753:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596672:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.417805:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.419240:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596672:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.419264:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b35180 x1875444569596416/t30064771097(30064771097) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.419326:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.419343:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.419357:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788572598.419368:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00000080:00200000:0.0:1788572598.419380:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.419386:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.419389:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.419395:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.419399:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 26 lru_type: 1 00010000:00010000:0.0:1788572598.419407:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1788572598.419418:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.419425:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.419431:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.419435:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.419438:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.419451:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.419456:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-12, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845cf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.419463:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-12, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.419481:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-12,[0x240002729:0x38:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.419536:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.419543:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.419552:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.419558:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b36d80 x1875444569596928/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.419568:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.419583:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b36d80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596928:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.419714:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5a80 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.422132:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b36d80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569596928:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.422153:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422184:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x38:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422193:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422203:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422213:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422226:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422234:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422249:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.422260:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b36d80 x1875444569596928/t30064771099(30064771099) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.422272:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b36d80 x1875444569596928/t30064771099(30064771099) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.422284:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.422295:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422304:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/56 (0 0) 00000080:00200000:0.0:1788572598.422310:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.422322:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-12, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.422351:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x38:0x0](ffff920a50d43248) 00000080:00200000:0.0:1788572598.422478:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x38:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.422483:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x38:0x0](ffff920a50d43248) 00000080:00010000:0.0:1788572598.422487:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x38:0x0](ffff920a50d43248) for lock 0x8364c713e25bb6d8 00000080:00010000:0.0:1788572598.422496:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x38:0x0](ffff920a50d43248) 00000080:00200008:0.0:1788572598.422508:0:166681:0:(file.c:1090:ll_file_open()) START file f-12:[0x240002729:0x38:0x0](ffff920a50d43248), flags 100102 00000002:00100000:0.0:1788572598.422529:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b36d80 x1875444569596928/t30064771099(30064771099) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.422565:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6d8 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.422571:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.422580:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.422634:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x38:0x0](ffff920a50d43248), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.422661:0:166681:0:(file.c:479:ll_file_release()) START file f-12:[0x240002729:0x38:0x0](ffff920a50d43248), flags 100002 00000002:00100000:0.0:1788572598.422698:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b36d80 x1875444569596928/t30064771099(30064771099) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.422709:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.422715:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34e00 x1875444569597184/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.422732:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597184:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.422801:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.425431:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34e00 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597184:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.425461:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b36d80 x1875444569596928/t30064771099(30064771099) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.425521:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.425539:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.425556:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1788572598.425570:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00000080:00200000:0.0:1788572598.425613:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.425618:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.425620:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.425626:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.425628:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 27 lru_type: 1 00010000:00010000:0.0:1788572598.425634:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:0.0:1788572598.425652:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.425659:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.425665:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.425669:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.425672:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.425683:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.425688:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-13, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845c600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.425695:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-13, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.425714:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-13,[0x240002729:0x39:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.425762:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.425770:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.425779:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.425785:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b34380 x1875444569597440/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.425795:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.425812:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597440:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.425959:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.428132:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b34380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597440:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.428154:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428178:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/0,1 mode: --/PR res: [0x240002729:0x39:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428185:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/0,1 mode: --/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428195:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428204:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428214:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428221:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428232:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.428241:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a45b34380 x1875444569597440/t30064771101(30064771101) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.428252:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34380 x1875444569597440/t30064771101(30064771101) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.428263:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.428274:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428282:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/57 (0 0) 00000080:00200000:0.0:1788572598.428288:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.428299:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-13, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.428324:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x39:0x0](ffff920a50d442c8) 00000080:00200000:0.0:1788572598.428385:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x39:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.428389:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x39:0x0](ffff920a50d442c8) 00000080:00010000:0.0:1788572598.428392:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x39:0x0](ffff920a50d442c8) for lock 0x8364c713e25bb6df 00000080:00010000:0.0:1788572598.428400:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x39:0x0](ffff920a50d442c8) 00000080:00200008:0.0:1788572598.428411:0:166681:0:(file.c:1090:ll_file_open()) START file f-13:[0x240002729:0x39:0x0](ffff920a50d442c8), flags 100102 00000002:00100000:0.0:1788572598.428427:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a45b34380 x1875444569597440/t30064771101(30064771101) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.428467:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6df from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.428471:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.428479:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.428497:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x39:0x0](ffff920a50d442c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.428523:0:166681:0:(file.c:479:ll_file_release()) START file f-13:[0x240002729:0x39:0x0](ffff920a50d442c8), flags 100002 00000002:00100000:0.0:1788572598.428556:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a45b34380 x1875444569597440/t30064771101(30064771101) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.428573:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.428577:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a45b35f80 x1875444569597696/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.428613:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597696:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.428676:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.429727:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a45b35f80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597696:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.429745:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a45b34380 x1875444569597440/t30064771101(30064771101) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.429792:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.429802:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.429811:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788572598.429820:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00000080:00200000:0.0:1788572598.429829:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.429834:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.429837:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.429842:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.429845:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 28 lru_type: 1 00010000:00010000:0.0:1788572598.429853:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1788572598.429864:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.429869:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.429874:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.429878:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.429881:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.429891:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.429910:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-14, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845dc80, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.429917:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-14, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.429935:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-14,[0x240002729:0x3a:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.429982:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.429990:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.429999:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.430004:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2c380 x1875444569597952/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.430014:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.430028:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2c380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597952:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.430122:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a442c5180 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.433170:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2c380 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569597952:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.433196:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433220:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433226:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433234:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433241:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433250:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433255:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433264:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.433272:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2c380 x1875444569597952/t30064771103(30064771103) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.433280:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2c380 x1875444569597952/t30064771103(30064771103) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.433289:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.433298:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433305:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/58 (0 0) 00000080:00200000:0.0:1788572598.433311:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.433321:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-14, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.433345:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3a:0x0](ffff920a50d45348) 00000080:00200000:0.0:1788572598.433400:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3a:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.433403:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3a:0x0](ffff920a50d45348) 00000080:00010000:0.0:1788572598.433405:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3a:0x0](ffff920a50d45348) for lock 0x8364c713e25bb6e6 00000080:00010000:0.0:1788572598.433414:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3a:0x0](ffff920a50d45348) 00000080:00200008:0.0:1788572598.433424:0:166681:0:(file.c:1090:ll_file_open()) START file f-14:[0x240002729:0x3a:0x0](ffff920a50d45348), flags 100102 00000002:00100000:0.0:1788572598.433439:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2c380 x1875444569597952/t30064771103(30064771103) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.433489:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6e6 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.433494:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.433503:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.433520:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3a:0x0](ffff920a50d45348), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.433543:0:166681:0:(file.c:479:ll_file_release()) START file f-14:[0x240002729:0x3a:0x0](ffff920a50d45348), flags 100002 00000002:00100000:0.0:1788572598.433580:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2c380 x1875444569597952/t30064771103(30064771103) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.433612:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.433617:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2ed80 x1875444569598208/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.433637:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2ed80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598208:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.433704:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.435849:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2ed80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598208:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.435877:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2c380 x1875444569597952/t30064771103(30064771103) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.435969:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.435985:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.436001:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:0.0:1788572598.436014:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00000080:00200000:0.0:1788572598.436028:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.436034:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.436038:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.436043:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.436048:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 29 lru_type: 1 00010000:00010000:0.0:1788572598.436056:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1788572598.436070:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.436076:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.436082:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.436086:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.436090:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.436102:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.436107:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-15, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845cf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.436114:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-15, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.436133:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-15,[0x240002729:0x3b:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.436179:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.436188:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.436199:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.436205:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2e300 x1875444569598464/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.436216:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.436233:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2e300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598464:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.436348:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba6e40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.438504:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2e300 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598464:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.438527:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438555:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438562:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438571:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438581:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438612:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438620:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438632:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.438641:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2e300 x1875444569598464/t30064771105(30064771105) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.438663:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2e300 x1875444569598464/t30064771105(30064771105) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.438677:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.438690:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438699:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/59 (0 0) 00000080:00200000:0.0:1788572598.438705:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.438722:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-15, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.438765:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3b:0x0](ffff920a50d45b88) 00000080:00200000:0.0:1788572598.438840:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3b:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.438845:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3b:0x0](ffff920a50d45b88) 00000080:00010000:0.0:1788572598.438848:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3b:0x0](ffff920a50d45b88) for lock 0x8364c713e25bb6ed 00000080:00010000:0.0:1788572598.438855:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3b:0x0](ffff920a50d45b88) 00000080:00200008:0.0:1788572598.438867:0:166681:0:(file.c:1090:ll_file_open()) START file f-15:[0x240002729:0x3b:0x0](ffff920a50d45b88), flags 100102 00000002:00100000:0.0:1788572598.438884:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2e300 x1875444569598464/t30064771105(30064771105) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.438956:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6ed from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.438968:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.438978:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.438998:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3b:0x0](ffff920a50d45b88), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.439036:0:166681:0:(file.c:479:ll_file_release()) START file f-15:[0x240002729:0x3b:0x0](ffff920a50d45b88), flags 100002 00000002:00100000:0.0:1788572598.439078:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2e300 x1875444569598464/t30064771105(30064771105) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.439088:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.439093:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2c700 x1875444569598720/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.439111:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2c700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598720:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.439172:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba60c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.440504:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2c700 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598720:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.440531:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2e300 x1875444569598464/t30064771105(30064771105) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.440625:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.440642:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.440656:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1788572598.440669:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00000080:00200000:0.0:1788572598.440681:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.440686:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.440688:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.440692:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.440695:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 30 lru_type: 1 00010000:00010000:0.0:1788572598.440700:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1788572598.440709:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.440714:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.440718:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.440720:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.440722:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.440739:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.440743:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-16, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845c600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.440750:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-16, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.440770:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-16,[0x240002729:0x3c:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.440812:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.440820:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.440830:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.440835:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2f100 x1875444569598976/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.440857:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.440873:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598976:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.440994:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51ba60c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.444479:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f100 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569598976:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.444506:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444537:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444545:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444557:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444568:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444581:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444610:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444623:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.444636:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2f100 x1875444569598976/t30064771107(30064771107) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.444650:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f100 x1875444569598976/t30064771107(30064771107) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.444663:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.444677:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444687:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/60 (0 0) 00000080:00200000:0.0:1788572598.444693:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.444706:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-16, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.444732:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3c:0x0](ffff920a50d421c8) 00000080:00200000:0.0:1788572598.444800:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3c:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.444806:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3c:0x0](ffff920a50d421c8) 00000080:00010000:0.0:1788572598.444809:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3c:0x0](ffff920a50d421c8) for lock 0x8364c713e25bb6f4 00000080:00010000:0.0:1788572598.444820:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3c:0x0](ffff920a50d421c8) 00000080:00200008:0.0:1788572598.444833:0:166681:0:(file.c:1090:ll_file_open()) START file f-16:[0x240002729:0x3c:0x0](ffff920a50d421c8), flags 100102 00000002:00100000:0.0:1788572598.444853:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f100 x1875444569598976/t30064771107(30064771107) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.444888:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6f4 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.444904:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.444925:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.444946:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3c:0x0](ffff920a50d421c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.444972:0:166681:0:(file.c:479:ll_file_release()) START file f-16:[0x240002729:0x3c:0x0](ffff920a50d421c8), flags 100002 00000002:00100000:0.0:1788572598.445014:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2f100 x1875444569598976/t30064771107(30064771107) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.445025:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.445030:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2d880 x1875444569599232/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.445051:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2d880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599232:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.445116:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.446516:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2d880 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599232:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.446539:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2f100 x1875444569598976/t30064771107(30064771107) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.446620:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.446636:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.446650:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:0.0:1788572598.446660:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00000080:00200000:0.0:1788572598.446670:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.446674:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.446677:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.446681:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.446683:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 31 lru_type: 1 00010000:00010000:0.0:1788572598.446689:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1788572598.446696:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.446700:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.446705:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.446707:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.446709:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.446717:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.446720:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-17, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845d800, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.446725:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-17, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.446738:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-17,[0x240002729:0x3d:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.446775:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.446783:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.446792:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.446798:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2ca80 x1875444569599488/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.446807:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.446823:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2ca80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599488:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.446945:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.449025:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2ca80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599488:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.449047:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449073:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449082:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449091:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449102:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449114:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449122:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449134:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.449145:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2ca80 x1875444569599488/t30064771109(30064771109) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.449158:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2ca80 x1875444569599488/t30064771109(30064771109) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.449171:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.449181:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449188:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/61 (0 0) 00000080:00200000:0.0:1788572598.449193:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.449203:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-17, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.449228:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3d:0x0](ffff920a50d400c8) 00000080:00200000:0.0:1788572598.449284:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3d:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.449288:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3d:0x0](ffff920a50d400c8) 00000080:00010000:0.0:1788572598.449290:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3d:0x0](ffff920a50d400c8) for lock 0x8364c713e25bb6fb 00000080:00010000:0.0:1788572598.449297:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3d:0x0](ffff920a50d400c8) 00000080:00200008:0.0:1788572598.449307:0:166681:0:(file.c:1090:ll_file_open()) START file f-17:[0x240002729:0x3d:0x0](ffff920a50d400c8), flags 100102 00000002:00100000:0.0:1788572598.449320:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2ca80 x1875444569599488/t30064771109(30064771109) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.449342:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb6fb from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.449346:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.449353:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.449373:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3d:0x0](ffff920a50d400c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.449400:0:166681:0:(file.c:479:ll_file_release()) START file f-17:[0x240002729:0x3d:0x0](ffff920a50d400c8), flags 100002 00000002:00100000:0.0:1788572598.449451:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2ca80 x1875444569599488/t30064771109(30064771109) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.449461:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.449466:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2df80 x1875444569599744/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.449484:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2df80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599744:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.449550:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.450952:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2df80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569599744:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.450971:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2ca80 x1875444569599488/t30064771109(30064771109) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.451026:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.451041:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.451056:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1788572598.451068:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00000080:00200000:0.0:1788572598.451078:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.451082:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.451084:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.451088:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.451091:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 32 lru_type: 1 00010000:00010000:0.0:1788572598.451097:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1788572598.451105:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.451110:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.451114:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.451116:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.451118:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.451128:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.451131:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-18, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845cf00, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.451136:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-18, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.451152:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-18,[0x240002729:0x3e:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.451193:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.451201:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.451210:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.451215:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2f800 x1875444569600000/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.451224:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.451240:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600000:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.451365:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.453498:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f800 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600000:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.453519:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453546:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453555:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453564:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453575:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453609:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453619:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453632:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.453644:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2f800 x1875444569600000/t30064771111(30064771111) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.453657:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f800 x1875444569600000/t30064771111(30064771111) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.453669:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.453681:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453691:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/62 (0 0) 00000080:00200000:0.0:1788572598.453697:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.453709:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-18, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.453735:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3e:0x0](ffff920a473ccb08) 00000080:00200000:0.0:1788572598.453803:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3e:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.453807:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3e:0x0](ffff920a473ccb08) 00000080:00010000:0.0:1788572598.453810:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3e:0x0](ffff920a473ccb08) for lock 0x8364c713e25bb702 00000080:00010000:0.0:1788572598.453817:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3e:0x0](ffff920a473ccb08) 00000080:00200008:0.0:1788572598.453828:0:166681:0:(file.c:1090:ll_file_open()) START file f-18:[0x240002729:0x3e:0x0](ffff920a473ccb08), flags 100102 00000002:00100000:0.0:1788572598.453842:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f800 x1875444569600000/t30064771111(30064771111) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.453874:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb702 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.453879:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.453888:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.453919:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3e:0x0](ffff920a473ccb08), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.453945:0:166681:0:(file.c:479:ll_file_release()) START file f-18:[0x240002729:0x3e:0x0](ffff920a473ccb08), flags 100002 00000002:00100000:0.0:1788572598.453976:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2f800 x1875444569600000/t30064771111(30064771111) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.453983:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.453986:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2d180 x1875444569600256/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.454001:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2d180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600256:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.454074:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.457169:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2d180 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600256:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.457204:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2f800 x1875444569600000/t30064771111(30064771111) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.457269:0:166681:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.457285:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.457299:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788572598.457309:0:166681:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00000080:00200000:0.0:1788572598.457319:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.457322:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.457325:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.457330:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301833bf8 00010000:00010000:0.0:1788572598.457332:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 33 lru_type: 1 00010000:00010000:0.0:1788572598.457338:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1788572598.457348:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.457353:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.457357:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.457359:0:166681:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.457361:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.457369:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.457372:0:166681:0:(namei.c:1412:ll_atomic_open()) VFS Op:name=f-19, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), file ffff920a5845c600, open_flags 8042, mode 81a4 opened 0 00000080:00200000:0.0:1788572598.457379:0:166681:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=f-19, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000002:00010000:0.0:1788572598.457396:0:166681:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: f-19,[0x240002729:0x3f:0x0]) in obj [0x240002729:0x2b:0x0], intent: open|creat flags 0100103 00010000:00010000:0.0:1788572598.457439:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.457445:0:166681:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.457451:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.457456:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2fb80 x1875444569600512/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/66320 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.457465:0:166681:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166681 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.457483:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2fb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600512:192.168.202.157@tcp:101:createmany.0 00000100:00100000:0.0:1788572598.457619:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.461003:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2fb80 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600512:192.168.202.157@tcp:101:createmany.0 00010000:00010000:0.0:1788572598.461032:0:166681:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/0,1 mode: --/CW res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461055:0:166681:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461061:0:166681:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/0,1 mode: --/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461069:0:166681:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461075:0:166681:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/0,1 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461094:0:166681:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461103:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CW) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 5/1,1 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461116:0:166681:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.461128:0:166681:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x3 disposition=0x37, status=0 req@ffff920a51a2fb80 x1875444569600512/t30064771113(30064771113) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.461137:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2fb80 x1875444569600512/t30064771113(30064771113) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.461147:0:166681:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open|creat, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.461156:0:166681:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461164:0:166681:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/63 (0 0) 00000080:00200000:0.0:1788572598.461169:0:166681:0:(namei.c:952:ll_intent_lock()) intent lock 3 on i1 [0x240002729:0x2b:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.461179:0:166681:0:(namei.c:1628:ll_create_it()) VFS Op:name=f-19, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08), intent=open|creat 00000080:00200000:0.0:1788572598.461200:0:166681:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x240002729:0x3f:0x0](ffff920a473ce3c8) 00000080:00200000:0.0:1788572598.461275:0:166681:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x3f:0x0]: layout version change: 4294967294 -> 0 00000080:00010000:0.0:1788572598.461280:0:166681:0:(namei.c:1594:ll_create_node()) setting l_ast_data to inode [0x240002729:0x3f:0x0](ffff920a473ce3c8) 00000080:00010000:0.0:1788572598.461284:0:166681:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x3f:0x0](ffff920a473ce3c8) for lock 0x8364c713e25bb709 00000080:00010000:0.0:1788572598.461294:0:166681:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x3f:0x0](ffff920a473ce3c8) 00000080:00200008:0.0:1788572598.461306:0:166681:0:(file.c:1090:ll_file_open()) START file f-19:[0x240002729:0x3f:0x0](ffff920a473ce3c8), flags 100102 00000002:00100000:0.0:1788572598.461322:0:166681:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2fb80 x1875444569600512/t30064771113(30064771113) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00010000:0.0:1788572598.461350:0:166681:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb709 from it ffff920a52f776c0 00010000:00010000:0.0:1788572598.461353:0:166681:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.461359:0:166681:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.461375:0:166681:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x3f:0x0](ffff920a473ce3c8), inode mode 81a4 mask 40 00000080:00200008:0.0:1788572598.461400:0:166681:0:(file.c:479:ll_file_release()) START file f-19:[0x240002729:0x3f:0x0](ffff920a473ce3c8), flags 100002 00000002:00100000:0.0:1788572598.461446:0:166681:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2fb80 x1875444569600512/t30064771113(30064771113) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.461456:0:166681:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.461461:0:166681:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2e680 x1875444569600768/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.461481:0:166681:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2e680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600768:192.168.202.157@tcp:35:createmany.0 00000100:00100000:0.0:1788572598.461542:0:166681:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.463123:0:166681:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2e680 pname:cluuid:pid:xid:nid:opc:job createmany:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166681:1875444569600768:192.168.202.157@tcp:35:createmany.0 00000002:00100000:0.0:1788572598.463147:0:166681:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2fb80 x1875444569600512/t30064771113(30064771113) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.469947:0:166687:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.469979:0:166687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.469996:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1788572598.470009:0:166687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00000080:00200000:0.0:1788572598.470019:0:166687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.470025:0:166687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.470028:0:166687:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.470033:0:166687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf73019ffbf8 00010000:00010000:0.0:1788572598.470035:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 34 lru_type: 1 00010000:00010000:0.0:1788572598.470042:0:166687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:0.0:1788572598.470054:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.470061:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=27 00000080:00200000:0.0:1788572598.470067:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.470072:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=27 00000080:00200000:0.0:1788572598.470075:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.470079:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=267 00000080:00200000:0.0:1788572598.470101:0:166687:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.470107:0:166687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.470112:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1788572598.470118:0:166687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00000080:00200000:0.0:1788572598.470122:0:166687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.470125:0:166687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.470127:0:166687:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.470129:0:166687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf73019ffbf8 00010000:00010000:0.0:1788572598.470131:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 35 lru_type: 1 00010000:00010000:0.0:1788572598.470136:0:166687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1788572598.470141:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.470144:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=27 00000080:00200000:0.0:1788572598.470146:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.470147:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=267 00000080:00200000:0.0:1788572598.470149:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 44 00000080:00200008:0.0:1788572598.470154:0:166687:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 304000 00000002:00010000:0.0:1788572598.470164:0:166687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x200027501:0x7:0x0], intent: open flags 0100240304001 00010000:00010000:0.0:1788572598.470202:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.470207:0:166687:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.470215:0:166687:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.470225:0:166687:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2f480 x1875444569601024/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.470231:0:166687:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.470246:0:166687:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601024:192.168.202.157@tcp:101:lfs.0 00000100:00100000:0.0:1788572598.470383:0:166687:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.471868:0:166687:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601024:192.168.202.157@tcp:101:lfs.0 00010000:00010000:0.0:1788572598.471887:0:166687:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471909:0:166687:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471916:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471923:0:166687:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x10206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471932:0:166687:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471939:0:166687:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60bc0800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:0.0:1788572598.471943:0:166687:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471950:0:166687:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 4/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x206400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471957:0:166687:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 5/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x20e400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471964:0:166687:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572598.471970:0:166687:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 3/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.471977:0:166687:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb710 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x1 rrc: 2 type: IBT flags: 0x4a0f400000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.472003:0:166687:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a51a2f480 x1875444569601024/t30064771115(30064771115) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.472017:0:166687:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f480 x1875444569601024/t30064771115(30064771115) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.472032:0:166687:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x7:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572598.472062:0:166687:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x2b:0x0], updating i_size 4096 00000002:00100000:0.0:1788572598.472077:0:166687:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f480 x1875444569601024/t30064771115(30064771115) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.472140:0:166687:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.472151:0:166687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.472159:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788572598.472168:0:166687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00000080:00200000:0.0:1788572598.472175:0:166687:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.472180:0:166687:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.472183:0:166687:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.472188:0:166687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf73019ffbf8 00010000:00010000:0.0:1788572598.472191:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 36 lru_type: 1 00010000:00010000:0.0:1788572598.472199:0:166687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788572598.472217:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.472222:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=25 00000080:00200000:0.0:1788572598.472225:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.472227:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=25 00000080:00200000:0.0:1788572598.472229:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.472232:0:166687:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=265 00000080:00200000:0.0:1788572598.472235:0:166687:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2d:0x0](ffff920a50d43a88), inode mode 81a4 mask 44 00000080:00200008:0.0:1788572598.472239:0:166687:0:(file.c:1090:ll_file_open()) START file f-1:[0x240002729:0x2d:0x0](ffff920a50d43a88), flags 100400 00000002:00010000:0.0:1788572598.472258:0:166687:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2d:0x0]) in obj [0x240002729:0x2b:0x0], intent: open flags 0240100401 00010000:00010000:0.0:1788572598.472294:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.472303:0:166687:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.472313:0:166687:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.472317:0:166687:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2ea00 x1875444569601280/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.472323:0:166687:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166687 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.472337:0:166687:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601280:192.168.202.157@tcp:101:lfs.0 00000100:00100000:0.0:1788572598.472458:0:166687:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4900 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.473883:0:166687:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2ea00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601280:192.168.202.157@tcp:101:lfs.0 00010000:00010000:0.0:1788572598.473916:0:166687:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473949:0:166687:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473957:0:166687:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473967:0:166687:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473978:0:166687:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473990:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473995:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.473999:0:166687:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.474006:0:166687:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a51a2ea00 x1875444569601280/t30064771116(30064771116) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.474038:0:166687:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2ea00 x1875444569601280/t30064771116(30064771116) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.474048:0:166687:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: open, lvb_len: 56 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:0.0:1788572598.474061:0:166687:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.474073:0:166687:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.474083:0:166687:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.474090:0:166687:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.474097:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.474103:0:166687:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.474130:0:166687:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:0.0:1788572598.474138:0:166687:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x240002729:0x2b:0x0] suppgids 0 0: rc 0 00010000:00010000:1.0:1788572598.474160:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572598.474166:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.474173:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60bc1800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572598.474177:0:160930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:0.0:1788572598.474182:0:166687:0:(file.c:6843:ll_layout_conf()) [0x240002729:0x2d:0x0]: layout version change: 0 -> 0 00010000:00010000:1.0:1788572598.474185:0:160930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00010000:0.0:1788572598.474189:0:166687:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2d:0x0](ffff920a50d43a88) for lock 0x8364c713e25bb68b 00000080:00010000:0.0:1788572598.474201:0:166687:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb68b from it ffffaf73019ffbf0 00010000:00010000:1.0:1788572598.474202:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.474208:0:166687:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.474219:0:166687:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572598.474229:0:160930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000002:00100000:0.0:1788572598.474244:0:166687:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2ea00 x1875444569601280/t30064771116(30064771116) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00010000:00010000:1.0:1788572598.474254:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 2/0,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.474260:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572598.474279:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.474284:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1800/0x8364c713e25bb717 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0161 expref: -99 pid: 166687 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572598.474297:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572598.474298:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572598.474300:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200080:0.0:1788572598.474309:0:166687:0:(file.c:4646:ll_file_ioctl()) VFS Op:inode=[0x240002729:0x2d:0x0](ffff920a50d43a88) cmd=800466af arg=7ffc45f1873c 00000100:00100000:1.0:1788572598.474310:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569601408:192.168.202.157@tcp:103:ldlm_bl.0 00000080:00200008:0.0:1788572598.474335:0:166687:0:(file.c:479:ll_file_release()) START file f-1:[0x240002729:0x2d:0x0](ffff920a50d43a88), flags 100000 00000002:00100000:0.0:1788572598.474366:0:166687:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2ea00 x1875444569601280/t30064771116(30064771116) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.474389:0:166687:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.474395:0:166687:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2dc00 x1875444569601664/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.474409:0:166687:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601664:192.168.202.157@tcp:35:lfs.0 00000100:00100000:0.0:1788572598.474446:0:166687:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.475618:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569601408:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572598.475627:0:166687:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601664:192.168.202.157@tcp:35:lfs.0 00000002:00100000:0.0:1788572598.475652:0:166687:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2ea00 x1875444569601280/t30064771116(30064771116) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788572598.475778:0:166687:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 304000 00000002:00100000:0.0:1788572598.475812:0:166687:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2f480 x1875444569601024/t30064771115(30064771115) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.475819:0:166687:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.475822:0:166687:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2dc00 x1875444569601920/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.475839:0:166687:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601920:192.168.202.157@tcp:35:lfs.0 00000100:00100000:0.0:1788572598.475871:0:166687:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.476992:0:166687:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166687:1875444569601920:192.168.202.157@tcp:35:lfs.0 00000002:00100000:0.0:1788572598.477011:0:166687:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2f480 x1875444569601024/t30064771115(30064771115) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.481083:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.481110:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.481128:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788572598.481142:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00000080:00200000:0.0:1788572598.481150:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.481157:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.481160:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.481168:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301507b58 00010000:00010000:0.0:1788572598.481171:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 37 lru_type: 1 00010000:00010000:0.0:1788572598.481180:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788572598.481193:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.481198:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=8 00000080:00200008:0.0:1788572598.481206:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572598.481209:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988),name=d80c.replay-single 00000002:00010000:0.0:1788572598.481218:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.481224:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.481230:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.481234:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.481236:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:0.0:1788572598.481238:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb66f 00000080:00010000:0.0:1788572598.481240:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb66f from it ffffaf7301507cb8 00010000:00010000:0.0:1788572598.481242:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.481247:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572598.481265:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.481269:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.481272:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788572598.481277:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00000080:00200000:0.0:1788572598.481281:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.481283:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.481284:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.481286:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301507bf8 00010000:00010000:0.0:1788572598.481288:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 38 lru_type: 1 00010000:00010000:0.0:1788572598.481293:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788572598.481297:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.481299:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=266 00000080:00200000:0.0:1788572598.481302:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572598.481305:0:166688:0:(file.c:1090:ll_file_open()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), flags 704400 00000002:00010000:0.0:1788572598.481314:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0100240704401 00010000:00010000:0.0:1788572598.481348:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.481353:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.481361:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.481370:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2dc00 x1875444569602176/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.481377:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.481392:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602176:192.168.202.157@tcp:101:rm.0 00000100:00100000:0.0:1788572598.481551:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.483480:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602176:192.168.202.157@tcp:101:rm.0 00010000:00010000:0.0:1788572598.483511:0:166688:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483528:0:166688:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483536:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483543:0:166688:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483552:0:166688:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483558:0:166688:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60bc0800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:0.0:1788572598.483563:0:166688:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483569:0:166688:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483576:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483583:0:166688:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:0.0:1788572598.483618:0:166688:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.483625:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb71e lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.483644:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a51a2dc00 x1875444569602176/t352187318277(352187318277) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.483668:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2dc00 x1875444569602176/t352187318277(352187318277) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.483685:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572598.483710:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000002:00100000:0.0:1788572598.483729:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2dc00 x1875444569602176/t352187318277(352187318277) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788572598.483785:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572598.483791:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988),name=d80c.replay-single 00000002:00010000:0.0:1788572598.483806:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.483819:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572598.483830:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:0.0:1788572598.483837:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.483843:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:0.0:1788572598.483846:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb66f 00000080:00010000:0.0:1788572598.483851:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb66f from it ffffaf7301507cd8 00010000:00010000:0.0:1788572598.483854:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572598.483862:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572598.483943:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) pos/size0/4096 32bit_api 0 00000002:00010000:0.0:1788572598.483954:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200027501:0x7:0x0], intent: readdir flags 0 00010000:00010000:0.0:1788572598.483960:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.483968:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000100:00100000:0.0:1788572598.484509:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602304:192.168.202.157@tcp:37:rm.0 00000100:00100000:0.0:1788572598.484567:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.491556:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602304:192.168.202.157@tcp:37:rm.0 00010000:00010000:0.0:1788572598.492226:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.492247:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:0.0:1788572598.492340:0:166688:0:(file.c:479:ll_file_release()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), flags 704000 00000002:00100000:0.0:1788572598.492407:0:166688:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2dc00 x1875444569602176/t352187318277(352187318277) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.492426:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.492432:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2f480 x1875444569602560/t0(0) o35->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.492454:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602560:192.168.202.157@tcp:35:rm.0 00000100:00100000:0.0:1788572598.492526:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.494207:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602560:192.168.202.157@tcp:35:rm.0 00000002:00100000:0.0:1788572598.494236:0:166688:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2dc00 x1875444569602176/t352187318277(352187318277) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 301/-1 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200000:0.0:1788572598.494322:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.494340:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.494354:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788572598.494368:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00000080:00200000:0.0:1788572598.494377:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.494383:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.494387:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.494393:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301507bf8 00010000:00010000:0.0:1788572598.494397:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 39 lru_type: 1 00010000:00010000:0.0:1788572598.494406:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788572598.494419:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.494426:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80c.replay-single, flags=266 00000080:00200000:0.0:1788572598.494434:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 44 00000080:00200008:0.0:1788572598.494441:0:166688:0:(file.c:1090:ll_file_open()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), flags 704400 00000002:00010000:0.0:1788572598.494455:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0104240704401 00010000:00010000:0.0:1788572598.494492:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.494500:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:0.0:1788572598.494510:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:0.0:1788572598.494514:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2f480 x1875444569602816/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:0.0:1788572598.494521:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.494534:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602816:192.168.202.157@tcp:101:rm.0 00000100:00100000:0.0:1788572598.494685:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.496656:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602816:192.168.202.157@tcp:101:rm.0 00010000:00010000:0.0:1788572598.496682:0:166688:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.496694:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.496706:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 4/1,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.496717:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 4/1,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.496730:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:0.0:1788572598.496742:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.496761:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 3/1,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.496771:0:166688:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 16 res 8590095617/7 (0 0) 00000080:00200000:0.0:1788572598.496776:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:0.0:1788572598.496808:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00010000:0.0:1788572598.496815:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb725 00000080:00010000:0.0:1788572598.496821:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb725 from it ffffaf7301507bd0 00010000:00010000:0.0:1788572598.496825:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 3/1,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.496835:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:0.0:1788572598.496856:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:0.0:1788572598.496935:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), request_mask 2047, flags 0, foreign 0 00000080:00200000:0.0:1788572598.496942:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988),name=d80c.replay-single 00000002:00010000:0.0:1788572598.496958:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0x7:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.496968:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572598.496978:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:0.0:1788572598.496986:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.496990:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:0.0:1788572598.496994:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb66f 00000080:00010000:0.0:1788572598.496997:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb66f from it ffffaf7301507cd8 00010000:00010000:0.0:1788572598.497001:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:0.0:1788572598.497008:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:0.0:1788572598.497084:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) pos/size0/4096 32bit_api 0 00000080:00200000:0.0:1788572598.497105:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:0.0:1788572598.497120:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.497127:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=0 00000080:00200008:0.0:1788572598.497134:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572598.497618:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08),name=remote_dir 00000002:00010000:2.0:1788572598.497638:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572598.497683:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.497692:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.497701:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572598.497721:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602944:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572598.497774:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.499153:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569602944:192.168.202.157@tcp:101:rm.0 00010000:00010000:2.0:1788572598.499171:0:166688:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499194:0:166688:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499201:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499211:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499219:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499228:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499241:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499258:0:166688:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572598.499270:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a60ac6680 x1875444569602944/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572598.499282:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499290:0:166688:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/43 (0 0) 00000080:00200000:2.0:1788572598.499296:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572598.499317:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x2b:0x0], updating i_size 4096 00000080:00010000:2.0:1788572598.499323:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb72c 00000080:00010000:2.0:1788572598.499327:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00010000:2.0:1788572598.499331:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb72c from it ffffaf7301507cb8 00010000:00010000:2.0:1788572598.499333:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499341:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572598.499400:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.499408:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:2.0:1788572598.499432:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572598.499445:0:166688:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 704400 00000002:00010000:2.0:1788572598.499464:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x200027501:0x7:0x0], intent: open flags 0104240704401 00010000:00010000:2.0:1788572598.499499:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.499506:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:2.0:1788572598.499514:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:2.0:1788572598.499520:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac6680 x1875444569603200/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:2.0:1788572598.499528:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572598.499541:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603200:192.168.202.157@tcp:101:rm.0 00000100:00100000:2.0:1788572598.499696:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.500933:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603200:192.168.202.157@tcp:101:rm.0 00010000:00010000:2.0:1788572598.500956:0:166688:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.500964:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.500970:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 4/1,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.500976:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 4/1,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1788572598.500988:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x200002b, status=0 req@ffff920a60ac6680 x1875444569603200/t30064771119(30064771119) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00100000:2.0:1788572598.500995:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac6680 x1875444569603200/t30064771119(30064771119) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1788572598.501003:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 3/1,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.501009:0:166688:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 16 res 9663686441/43 (0 0) 00000080:00200000:2.0:1788572598.501013:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200027501:0x7:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1788572598.501039:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x2b:0x0], updating i_size 4096 00000080:00010000:2.0:1788572598.501044:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb733 00000080:00010000:2.0:1788572598.501050:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb733 from it ffffaf7301507bd0 00010000:00010000:2.0:1788572598.501053:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 3/1,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1788572598.501061:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000002:00100000:2.0:1788572598.501076:0:166688:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a60ac6680 x1875444569603200/t30064771119(30064771119) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 2 fl Complete:RPQU/604/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000080:00200008:2.0:1788572598.501127:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572598.501133:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08),name=remote_dir 00000002:00010000:2.0:1788572598.501148:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572598.501157:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.501176:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572598.501187:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572598.501191:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00010000:2.0:1788572598.501194:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb72c 00000080:00010000:2.0:1788572598.501198:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb72c from it ffffaf7301507cd8 00010000:00010000:2.0:1788572598.501201:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.501209:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572598.501233:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08) pos/size0/4096 32bit_api 0 00000002:00010000:2.0:1788572598.501249:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x240002729:0x2b:0x0], intent: readdir flags 0 00010000:00010000:2.0:1788572598.501254:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572598.501260:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000100:00100000:2.0:1788572598.501886:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603328:192.168.202.157@tcp:37:rm.0 00000100:00100000:2.0:1788572598.501963:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.505002:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603328:192.168.202.157@tcp:37:rm.0 00010000:00010000:2.0:1788572598.505560:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1788572598.505572:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200008:2.0:1788572598.505635:0:166688:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 704000 00010000:00010000:2.0:1788572598.505658:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1788572598.505691:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.505696:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=remote_dir, flags=258 00000080:00200000:2.0:1788572598.505702:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 44 00000080:00200008:2.0:1788572598.505707:0:166688:0:(file.c:1090:ll_file_open()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 704400 00000080:00200008:2.0:1788572598.505718:0:166688:0:(file.c:6296:ll_getattr_dentry()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1788572598.505722:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08),name=remote_dir 00000002:00010000:2.0:1788572598.505736:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x240002729:0x2b:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 0 00010000:00010000:2.0:1788572598.505742:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572598.505748:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1788572598.505753:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1788572598.505756:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00010000:2.0:1788572598.505758:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb72c 00000080:00010000:2.0:1788572598.505762:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb72c from it ffffaf7301507cd8 00010000:00010000:2.0:1788572598.505764:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1788572598.505772:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1788572598.505789:0:166688:0:(llite_lib.c:2903:ll_statfs()) VFS Op:sb=lustre (ffff920a47b93800) remote_dir proj=0 00000080:00000004:2.0:1788572598.505792:0:166688:0:(obd_class.h:1090:obd_statfs()) lustre-clilmv-ffff920a47b93800: age 9225, max_age 9356 00800000:00000004:2.0:1788572598.505798:0:166688:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: age 9356, max_age 9356 00800000:00000004:2.0:1788572598.505799:0:166688:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0000-mdc-ffff920a47b93800: use ffff920a46419e18 cache blocks 317221/353529 objects 1023635/1024000 00800000:00000004:2.0:1788572598.505801:0:166688:0:(obd_class.h:1090:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: age 9356, max_age 9356 00800000:00000004:2.0:1788572598.505803:0:166688:0:(obd_class.h:1130:obd_statfs()) lustre-MDT0001-mdc-ffff920a47b93800: use ffff920a46418458 cache blocks 317321/353529 objects 1023722/1024000 00000080:00000004:2.0:1788572598.505806:0:166688:0:(obd_class.h:1116:obd_statfs()) lustre-clilmv-ffff920a47b93800: update ffff920a4641b7d8 cache blocks 634542/707058 objects 2047357/2048000 00000080:00000004:2.0:1788572598.505808:0:166688:0:(llite_lib.c:2755:ll_statfs_internal()) MDC blocks 634542/707058 objects 2047357/2048000 00000080:00000004:2.0:1788572598.505810:0:166688:0:(obd_class.h:1090:obd_statfs()) lustre-clilov-ffff920a47b93800: age 9225, max_age 9356 00020000:00000004:2.0:1788572598.505819:0:166688:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0000-osc-ffff920a47b93800: age 9356, max_age 9356 00000008:00000004:2.0:1788572598.505821:0:166688:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0000-osc-ffff920a47b93800: use ffff920a51525198 cache blocks 901356/958279 objects 261711/262144 00020000:00000004:2.0:1788572598.505825:0:166688:0:(obd_class.h:1056:obd_statfs_async()) lustre-OST0001-osc-ffff920a47b93800: age 9356, max_age 9356 00000008:00000004:2.0:1788572598.505826:0:166688:0:(osc_request.c:3417:osc_statfs_async()) lustre-OST0001-osc-ffff920a47b93800: use ffff920a515237d8 cache blocks 901360/958279 objects 261727/262144 00000080:00000004:2.0:1788572598.505832:0:166688:0:(obd_class.h:1116:obd_statfs()) lustre-clilov-ffff920a47b93800: update ffff920a4641d198 cache blocks 1802716/1916558 objects 523438/524288 00000080:00000004:2.0:1788572598.505834:0:166688:0:(llite_lib.c:2766:ll_statfs_internal()) OSC blocks 1802716/1916558 objects 523438/524288 00000080:00200000:2.0:1788572598.505843:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08) pos/size0/4096 32bit_api 0 00000080:00200000:2.0:1788572598.505907:0:166688:0:(dir.c:332:ll_iterate()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08) pos/size18446744073709551614/4096 32bit_api 0 00000080:00200000:2.0:1788572598.505919:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.505925:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-13, flags=0 00000080:00200000:2.0:1788572598.505930:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572598.505934:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-13, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:2.0:1788572598.505957:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572598.506017:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572598.506026:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x39:0x0]: losing layout lock 00010000:00010000:2.0:1788572598.506050:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 1/0,0 mode: --/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.506054:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1788572598.506058:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1788572598.506060:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71a00/0x8364c713e25bb6df lrc: 0/0,0 mode: --/PR res: [0x240002729:0x39:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c006c expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.506070:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47816400/0x8364c713e25bb72c lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0199 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000020:00100000:2.0:1788572598.506080:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572598.506086:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac7480 x1875444569603584/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572598.506100:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603584:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572598.506148:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.509316:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603584:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572598.509377:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x39:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572598.509391:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572598.509428:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x39:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572598.509440:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x39:0x0](ffff920a50d442c8) 00000080:00200000:2.0:1788572598.509513:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.509524:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-2, flags=0 00000080:00200000:2.0:1788572598.509530:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572598.509535:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-2, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:2.0:1788572598.509571:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572598.509630:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x2e:0x0]: losing layout lock 00010000:00010000:2.0:1788572598.509675:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.509684:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572598.509688:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a46ec6000/0x8364c713e25bb692 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfeeb expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572598.509710:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572598.509717:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5f80 x1875444569603840/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572598.509738:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603840:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572598.509799:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.511639:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569603840:192.168.202.157@tcp:36:rm.0 00000080:00200000:2.0:1788572598.511680:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x2e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1788572598.511692:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1788572598.511716:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x2e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1788572598.511723:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2e:0x0](ffff920a50d3a1c8) 00000080:00200000:2.0:1788572598.511787:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:2.0:1788572598.511796:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-7, flags=0 00000080:00200000:2.0:1788572598.511800:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:2.0:1788572598.511804:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-7, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:2.0:1788572598.511837:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1788572598.511861:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x33:0x0]: losing layout lock 00010000:00010000:2.0:1788572598.511907:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1788572598.511914:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1788572598.511916:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79a00/0x8364c713e25bb6b5 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x33:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff9a expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1788572598.511942:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1788572598.511949:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a60ac5880 x1875444569604096/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572598.511967:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604096:192.168.202.157@tcp:36:rm.0 00000100:00100000:2.0:1788572598.512017:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a52645600 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.513959:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604096:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.513998:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x33:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.514008:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.514026:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x33:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.514032:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x33:0x0](ffff920a50d48908) 00000080:00200000:0.0:1788572598.514106:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.514116:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-18, flags=0 00000080:00200000:0.0:1788572598.514121:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.514125:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-18, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.514155:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.514187:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3e:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.514219:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.514225:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.514227:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71000/0x8364c713e25bb702 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3e:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c011b expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.514242:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.514247:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2dc00 x1875444569604352/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.514260:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604352:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.514302:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4540 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.516129:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2dc00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604352:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.516179:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.516188:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.516205:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.516210:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3e:0x0](ffff920a473ccb08) 00000080:00200000:0.0:1788572598.516276:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.516285:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-3, flags=0 00000080:00200000:0.0:1788572598.516289:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.516293:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-3, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.516324:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.516340:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x2f:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.516384:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.516392:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.516396:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70a00/0x8364c713e25bb699 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff0e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.516415:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.516429:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2ea00 x1875444569604608/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.516447:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2ea00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604608:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.516509:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.518676:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2ea00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604608:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.518723:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x2f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.518736:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.518760:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x2f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.518766:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2f:0x0](ffff920a50d41148) 00000080:00200000:0.0:1788572598.518831:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.518841:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-16, flags=0 00000080:00200000:0.0:1788572598.518846:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.518851:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-16, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.518881:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.518922:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3c:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.518959:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.518967:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.518971:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70c00/0x8364c713e25bb6f4 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00d5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.518995:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.519002:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2d500 x1875444569604864/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.519229:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2d500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604864:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.519290:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.521017:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2d500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569604864:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.521058:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.521069:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.521104:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.521112:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3c:0x0](ffff920a50d421c8) 00000080:00200000:0.0:1788572598.521173:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.521183:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-4, flags=0 00000080:00200000:0.0:1788572598.521188:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.521193:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-4, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.521223:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.521241:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x30:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.521271:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.521278:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.521281:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71600/0x8364c713e25bb6a0 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x30:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff31 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.521302:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.521310:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2c000 x1875444569605120/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.521329:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2c000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605120:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.521383:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.523098:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2c000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605120:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.523141:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x30:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.523151:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.523177:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x30:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.523183:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x30:0x0](ffff920a50d4a1c8) 00000080:00200000:0.0:1788572598.523251:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.523261:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-15, flags=0 00000080:00200000:0.0:1788572598.523266:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.523271:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-15, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.523302:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.523329:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3b:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.523360:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.523368:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.523370:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70200/0x8364c713e25bb6ed lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3b:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00b2 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.523391:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.523397:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a51a2ce00 x1875444569605376/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.523419:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2ce00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605376:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.523483:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.525275:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2ce00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605376:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.525323:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.525332:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.525353:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.525360:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3b:0x0](ffff920a50d45b88) 00000080:00200000:0.0:1788572598.525420:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.525446:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-0, flags=0 00000080:00200000:0.0:1788572598.525451:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.525455:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-0, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.525486:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.525503:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x2c:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.525538:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.525544:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.525547:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71800/0x8364c713e25bb684 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2c:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfea5 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.525567:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.525574:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a2d80 x1875444569605632/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.525611:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a2d80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605632:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.525666:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.527246:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a2d80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605632:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.527287:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x2c:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.527297:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.527316:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x2c:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.527322:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2c:0x0](ffff920a50d463c8) 00000080:00200000:0.0:1788572598.527388:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.527398:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-8, flags=0 00000080:00200000:0.0:1788572598.527402:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.527407:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-8, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.527445:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.527461:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x34:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.527489:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 1/0,0 mode: --/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.527497:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.527499:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79e00/0x8364c713e25bb6bc lrc: 0/0,0 mode: --/PR res: [0x240002729:0x34:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bffbd expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.527519:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.527525:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1880 x1875444569605888/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.527542:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605888:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.527660:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.529233:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1880 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569605888:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.529269:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x34:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.529279:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.529298:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x34:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.529304:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x34:0x0](ffff920a50d40908) 00000080:00200000:0.0:1788572598.529352:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.529361:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-9, flags=0 00000080:00200000:0.0:1788572598.529365:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.529369:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-9, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.529397:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.529409:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x35:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.529436:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.529443:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.529446:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70000/0x8364c713e25bb6c3 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x35:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bffe0 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.529465:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.529473:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a2300 x1875444569606144/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.529489:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a2300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606144:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.529541:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.531217:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a2300 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606144:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.531250:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x35:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.531260:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.531279:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x35:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.531286:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x35:0x0](ffff920a50d47448) 00000080:00200000:0.0:1788572598.531365:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.531374:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-11, flags=0 00000080:00200000:0.0:1788572598.531378:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.531383:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-11, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.531409:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.531423:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x37:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.531454:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.531461:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.531463:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71e00/0x8364c713e25bb6d1 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x37:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0026 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.531484:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.531492:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a0000 x1875444569606400/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.531515:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606400:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.531560:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4840 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.533190:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0000 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606400:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.533226:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x37:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.533235:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.533250:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x37:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.533255:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x37:0x0](ffff920a50d42a08) 00000080:00200000:0.0:1788572598.533341:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.533350:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-19, flags=0 00000080:00200000:0.0:1788572598.533354:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.533359:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-19, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.533395:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.533411:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3f:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.533460:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.533468:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.533471:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70600/0x8364c713e25bb709 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3f:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c013e expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.533491:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.533498:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a3100 x1875444569606656/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.533526:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3100 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606656:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.533578:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.535350:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3100 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606656:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.535392:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3f:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.535402:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.535421:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3f:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.535427:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3f:0x0](ffff920a473ce3c8) 00000080:00200000:0.0:1788572598.535476:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.535484:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-17, flags=0 00000080:00200000:0.0:1788572598.535487:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.535490:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-17, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.535512:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.535523:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3d:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.535547:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.535551:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.535553:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70e00/0x8364c713e25bb6fb lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c00f8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.535566:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.535570:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1c00 x1875444569606912/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.535582:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606912:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.535648:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4480 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.537138:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1c00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569606912:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.537175:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.537184:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.537201:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.537207:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3d:0x0](ffff920a50d400c8) 00000080:00200000:0.0:1788572598.537264:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.537271:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-1, flags=0 00000080:00200000:0.0:1788572598.537275:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.537279:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-1, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.537306:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572598.537322:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x2d:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.537351:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.537357:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.537360:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70400/0x8364c713e25bb68b lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2d:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bfec8 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788572598.537378:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.537385:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a2680 x1875444569607168/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.537407:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a2680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607168:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.537460:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.542065:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a2680 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607168:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.542118:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x2d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.542130:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.542155:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x2d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.542163:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2d:0x0](ffff920a50d43a88) 00000080:00200000:0.0:1788572598.542236:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.542247:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-10, flags=0 00000080:00200000:0.0:1788572598.542252:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.542256:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-10, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.542291:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.542308:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x36:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.542349:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 1/0,0 mode: --/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.542357:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.542360:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a70800/0x8364c713e25bb6ca lrc: 0/0,0 mode: --/PR res: [0x240002729:0x36:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0003 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.542382:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.542389:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a0700 x1875444569607424/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.542409:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607424:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.542480:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.544294:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0700 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607424:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.544331:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x36:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.544340:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.544357:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x36:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.544363:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x36:0x0](ffff920a50d41988) 00000080:00200000:0.0:1788572598.544414:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.544421:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-14, flags=0 00000080:00200000:0.0:1788572598.544424:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.544428:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-14, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.544457:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.544472:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x3a:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.544502:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.544511:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.544514:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71400/0x8364c713e25bb6e6 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x3a:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c008f expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.544528:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.544533:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a3480 x1875444569607680/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.544546:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607680:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.544725:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4b40 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.546673:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3480 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607680:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.546720:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x3a:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.546739:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.546759:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x3a:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.546767:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x3a:0x0](ffff920a50d45348) 00000080:00200000:0.0:1788572598.546870:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.546881:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-5, flags=0 00000080:00200000:0.0:1788572598.546886:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.546900:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-5, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.546929:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.546944:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x31:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.546978:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.546986:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.546990:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a79600/0x8364c713e25bb6a7 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x31:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff54 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.547007:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.547012:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1180 x1875444569607936/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.547026:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607936:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.547097:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.549186:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1180 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569607936:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.549235:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x31:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.549247:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.549271:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x31:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.549279:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x31:0x0](ffff920a50d49148) 00000080:00200000:0.0:1788572598.549348:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.549359:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-12, flags=0 00000080:00200000:0.0:1788572598.549364:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.549369:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-12, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.549402:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.549418:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x38:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.549454:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.549463:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.549466:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a51a71200/0x8364c713e25bb6d8 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x38:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109c0049 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.549500:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.549508:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1500 x1875444569608192/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.549529:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608192:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.549682:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.551550:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1500 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608192:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.551630:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x38:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.551642:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.551663:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x38:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.551670:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x38:0x0](ffff920a50d43248) 00000080:00200000:0.0:1788572598.551728:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.551738:0:166688:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=f-6, flags=0 00000080:00200000:0.0:1788572598.551743:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.551747:0:166688:0:(namei.c:2351:ll_unlink()) VFS Op:name=f-6, dir=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00010000:00010000:0.0:1788572598.551776:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.551802:0:166688:0:(vvp_object.c:117:vvp_conf_set()) [0x240002729:0x32:0x0]: losing layout lock 00010000:00010000:0.0:1788572598.551835:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 1/0,0 mode: --/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.551842:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.551845:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60a78600/0x8364c713e25bb6ae lrc: 0/0,0 mode: --/PR res: [0x240002729:0x32:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0x265cfe11109bff77 expref: -99 pid: 166681 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:0.0:1788572598.551864:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.551872:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a2a00 x1875444569608448/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 488/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.551903:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a2a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608448:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.551956:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.553806:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a2a00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608448:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.553846:0:166688:0:(vvp_io.c:1826:vvp_io_init()) [0x240002729:0x32:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:0.0:1788572598.553856:0:166688:0:(lov_io.c:1085:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:0.0:1788572598.553873:0:166688:0:(vvp_io.c:304:vvp_io_fini()) [0x240002729:0x32:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:0.0:1788572598.553879:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x32:0x0](ffff920a50d4ba88) 00000080:00200008:0.0:1788572598.553953:0:166688:0:(file.c:479:ll_file_release()) START file remote_dir:[0x240002729:0x2b:0x0](ffff920a50d44b08), flags 704000 00010000:00010000:0.0:1788572598.553975:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572598.554007:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.554021:0:166688:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), flags=0 00000080:00200000:0.0:1788572598.554026:0:166688:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988), intent=getattr 00000002:00010000:0.0:1788572598.554057:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0x7:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572598.554105:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x0/0x0 rrc: 4 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.554114:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.554123:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.554140:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608576:192.168.202.157@tcp:101:rm.0 00000100:00100000:0.0:1788572598.554194:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e46c0 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.555568:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608576:192.168.202.157@tcp:101:rm.0 00010000:00010000:0.0:1788572598.555581:0:166688:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555614:0:166688:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555620:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555628:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555636:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555643:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555648:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555653:0:166688:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.555659:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a479a0e00 x1875444569608576/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.555668:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555675:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.555681:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 2/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.555685:0:166688:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555690:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555694:0:166688:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555708:0:166688:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000002:00010000:0.0:1788572598.555716:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x0:0x0:0x0]) in obj [0x240002729:0x2b:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572598.555735:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.555739:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.555742:0:160929:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:0.0:1788572598.555744:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 3/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.555751:0:160929:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.555756:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608704:192.168.202.157@tcp:101:rm.0 00010000:00010000:3.0:1788572598.555766:0:160929:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60bc1a00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572598.555771:0:160929:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.555785:0:160929:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.555793:0:160929:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 3/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.555795:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00010000:00010000:3.0:1788572598.555814:0:160929:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572598.555843:0:160929:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 2/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572598.555851:0:160929:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572598.555877:0:160929:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572598.555882:0:2399:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00010000:00010000:3.0:1788572598.555903:0:160929:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1a00/0x8364c713e25bb73a lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c03fa expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1788572598.555909:0:2399:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569608832:192.168.202.157@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572598.555918:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572598.555964:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572598.555967:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572598.558852:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2399:1875444569608832:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572598.559094:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569608704:192.168.202.157@tcp:101:rm.0 00010000:00010000:0.0:1788572598.559109:0:166688:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 4/1,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559117:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 4/1,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559125:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559132:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559140:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559145:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 5/2,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559149:0:166688:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 4/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.559156:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a479a0a80 x1875444569608704/t0(0) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.559164:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559172:0:166688:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a53621400 type 13 mode 4 res 9663686441/43 (0 0) 00000080:00200000:0.0:1788572598.559182:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0x7:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.559206:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x240002729:0x2b:0x0], updating i_size 4096 00000080:00200000:0.0:1788572598.559208:0:166688:0:(namei.c:147:ll_iget()) got inode: [0x240002729:0x2b:0x0](ffff920a50d44b08): rc = 0 00000080:00010000:0.0:1788572598.559212:0:166688:0:(llite_internal.h:1981:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for remote lock 0x8364c713e25bb741 00000080:00010000:0.0:1788572598.559216:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) for lock 0x8364c713e25bb676 00000080:00010000:0.0:1788572598.559222:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00010000:0.0:1788572598.559225:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb676 from it ffffaf7301507e00 00010000:00010000:0.0:1788572598.559227:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.559233:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00010000:0.0:1788572598.559242:0:166688:0:(dcache.c:164:ll_intent_drop_lock()) releasing remote lock with cookie 0x8364c713e25bb741 from it ffffaf7301507e00 00010000:00010000:0.0:1788572598.559243:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 3/1,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.559248:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.559260:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.559266:0:166688:0:(namei.c:2261:ll_rmdir()) VFS Op:name=remote_dir, dir=[0x200027501:0x7:0x0](ffff920a50d49988) 00010000:00010000:0.0:1788572598.559284:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572598.559309:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572598.559320:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:0.0:1788572598.559341:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a51a71c00/0x8364c713e25bb676 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bfe6d expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:0.0:1788572598.559366:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 2/0,0 mode: PR/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000100:00100000:1.0:1788572598.559370:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569608960:192.168.202.157@tcp:103:rm.0 00010000:00010000:0.0:1788572598.559378:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 2/0,0 mode: CR/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:0.0:1788572598.559406:0:166688:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a60ac6680 x1875444569603200/t30064771119(30064771119) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.559413:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.559696:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a0e00 x1875444569609216/t0(0) o35->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.559713:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609216:192.168.202.157@tcp:35:rm.0 00000100:00100000:0.0:1788572598.559753:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00000100:00100000:1.0:1788572598.560304:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569608960:192.168.202.157@tcp:103:rm.0 00000100:00100000:0.0:1788572598.560774:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609216:192.168.202.157@tcp:35:rm.0 00000002:00100000:0.0:1788572598.560789:0:166688:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a60ac6680 x1875444569603200/t30064771119(30064771119) o101->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788572598.560819:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 1/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.560826:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 1/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.560833:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:0.0:1788572598.560835:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb741 lrc: 0/0,0 mode: --/PR res: [0x240002729:0x2b:0x0].0x0 bits 0x12/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0408 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.560841:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0001-mdc-ffff920a47b93800 lock: ffff920a47817200/0x8364c713e25bb733 lrc: 0/0,0 mode: --/CR res: [0x240002729:0x2b:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c01a7 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000020:00100000:0.0:1788572598.560853:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.560858:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a0e00 x1875444569609472/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 496/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.560871:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609472:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.560933:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00000100:00100000:2.0:1788572598.562994:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572598.563044:0:160921:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584516352 00000100:00100000:3.0:1788572598.563053:0:160921:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584516352 00000100:00100000:3.0:1788572598.563100:0:160921:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 11 00000100:00100000:3.0:1788572598.563112:0:160921:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a60ac6680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584516352:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572598.563128:0:160921:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572598.563176:0:160921:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572598.563182:0:160921:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a60ac6680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_000:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584516352:12345-192.168.202.157@tcp:104: Request processed in 73us (194us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572598.563190:0:160921:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 11 00010000:00010000:1.0:1788572598.563214:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572598.563224:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563233:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a47816800 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572598.563239:0:160930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563248:0:160930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563257:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 3/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563280:0:160930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572598.563322:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 2/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563329:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572598.563358:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 1/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000100:00100000:0.0:1788572598.563391:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569609600:192.168.202.157@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1788572598.563404:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47816800/0x8364c713e25bb66f lrc: 0/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109bfe51 expref: -99 pid: 166675 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788572598.563418:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572598.563421:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572598.563423:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572598.564794:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569609600:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572598.579085:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0e00 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609472:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.579130:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x240002729:0x2b:0x0](ffff920a50d44b08) 00000080:00200008:0.0:1788572598.579190:0:166688:0:(file.c:479:ll_file_release()) START file d80c.replay-single:[0x200027501:0x7:0x0](ffff920a50d49988), flags 704000 00010000:00010000:0.0:1788572598.579230:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:0.0:1788572598.579270:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.579286:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.579295:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788572598.579305:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00000080:00200000:0.0:1788572598.579311:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.579318:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.579322:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.579328:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301507c10 00010000:00010000:0.0:1788572598.579332:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 40 lru_type: 1 00010000:00010000:0.0:1788572598.579342:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1788572598.579359:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:0.0:1788572598.579375:0:166688:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=0 00000080:00200000:0.0:1788572598.579380:0:166688:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:0.0:1788572598.579391:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80c.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:0.0:1788572598.579425:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.579430:0:166688:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.579437:0:166688:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 166688 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:0.0:1788572598.579452:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609728:192.168.202.157@tcp:101:rm.0 00000100:00100000:0.0:1788572598.579517:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.581120:0:166688:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 352187318279 gen 1 00000100:00100000:0.0:1788572598.581134:0:166688:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000100:00100000:0.0:1788572598.581145:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609728:192.168.202.157@tcp:101:rm.0 00010000:00010000:0.0:1788572598.581158:0:166688:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581168:0:166688:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581174:0:166688:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581181:0:166688:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581189:0:166688:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581201:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581209:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581216:0:166688:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:0.0:1788572598.581226:0:166688:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a479a1f80 x1875444569609728/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'rm.0' uid:0 gid:0 projid:0 00000002:00010000:0.0:1788572598.581239:0:166688:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581247:0:166688:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/7 (0 0) 00000080:00200000:0.0:1788572598.581252:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:0.0:1788572598.581277:0:166688:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0x7:0x0], updating i_size 4096 00000080:00200000:0.0:1788572598.581280:0:166688:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0x7:0x0](ffff920a50d49988): rc = 0 00000080:00010000:0.0:1788572598.581286:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) for lock 0x8364c713e25bb748 00000080:00010000:0.0:1788572598.581294:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0x7:0x0](ffff920a50d49988) 00000080:00010000:0.0:1788572598.581298:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb748 from it ffffaf7301507e00 00010000:00010000:0.0:1788572598.581300:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:0.0:1788572598.581308:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:0.0:1788572598.581344:0:166688:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:0.0:1788572598.581369:0:166688:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:0.0:1788572598.581376:0:166688:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788572598.581394:0:166688:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00000080:00200000:0.0:1788572598.581410:0:166688:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:0.0:1788572598.581415:0:166688:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:0.0:1788572598.581419:0:166688:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb661 00000080:00010000:0.0:1788572598.581423:0:166688:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb661 from it ffffaf7301507d88 00010000:00010000:0.0:1788572598.581426:0:166688:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 41 lru_type: 1 00010000:00010000:0.0:1788572598.581433:0:166688:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000080:00200000:0.0:1788572598.581440:0:166688:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 3 00000080:00200000:0.0:1788572598.581446:0:166688:0:(namei.c:2261:ll_rmdir()) VFS Op:name=d80c.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08) 00010000:00010000:0.0:1788572598.581464:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:0.0:1788572598.581483:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.581499:0:166688:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 2/0,0 mode: CR/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000002:00100000:0.0:1788572598.581531:0:166688:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00000020:00100000:0.0:1788572598.581541:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:0.0:1788572598.581546:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1f80 x1875444569609984/t0(0) o35->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00100000:0.0:1788572598.581568:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609984:192.168.202.157@tcp:35:rm.0 00000100:00100000:0.0:1788572598.581642:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.582751:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569609984:192.168.202.157@tcp:35:rm.0 00000002:00100000:0.0:1788572598.582767:0:166688:0:(mdc_request.c:832:mdc_free_open()) @@@ free open request, rq_replay=0 req@ffff920a51a2f480 x1875444569602816/t352187318279(352187318279) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572614 ref 1 fl Complete:RQU/604/ffffffff rc 0/-1 job:'rm.0' uid:0 gid:0 projid:0 00010000:00010000:0.0:1788572598.582806:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 1/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.582812:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 1/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.582816:0:166688:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00010000:00010000:0.0:1788572598.582819:0:166688:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 3 locks packed 00010000:00010000:0.0:1788572598.582821:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc1000/0x8364c713e25bb748 lrc: 0/0,0 mode: --/PR res: [0x200027501:0x7:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0455 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:0.0:1788572598.582828:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60bc0800/0x8364c713e25bb725 lrc: 0/0,0 mode: --/CR res: [0x200027501:0x7:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0184 expref: -99 pid: 166688 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:0.0:1788572598.582837:0:166688:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a47817000/0x8364c713e25bb661 lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109bfbb1 expref: -99 pid: 165380 timeout: 0 lvb_type: 0 lru_score: 42 lru_type: 1 00000020:00100000:0.0:1788572598.582850:0:166688:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:0.0:1788572598.582855:0:166688:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a479a1f80 x1875444569610240/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 512/624 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572598.582870:0:166688:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569610240:192.168.202.157@tcp:36:rm.0 00000100:00100000:0.0:1788572598.582925:0:166688:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a519e4300 going to sleep for 11 seconds 00000100:00100000:0.0:1788572598.584632:0:166688:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job rm:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:166688:1875444569610240:192.168.202.157@tcp:36:rm.0 00000080:00200000:0.0:1788572598.584673:0:166688:0:(llite_lib.c:2066:ll_clear_inode()) VFS Op:inode=[0x200027501:0x7:0x0](ffff920a50d49988) 00010000:00010000:3.0:1788572598.683657:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572599.708659:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572600.731719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000001:02000400:1.0:1788572600.926465:0:167254:0:(debug.c:653:libcfs_debug_mark_buffer()) DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 21:43:20 (1788572600) 00000080:00200000:3.0:1788572600.945178:0:167257:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572600.945204:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572600.945261:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.945272:0:167257:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.945282:0:167257:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572600.945305:0:167257:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610368:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572600.945378:0:167257:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572600.946720:0:167257:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610368:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572600.946736:0:167257:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946747:0:167257:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946760:0:167257:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946768:0:167257:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946775:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946780:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946785:0:167257:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572600.946792:0:167257:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a529b0380 x1875444569610368/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572616 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:3.0:1788572600.946813:0:167257:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946820:0:167257:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:3.0:1788572600.946824:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:3.0:1788572600.946840:0:167257:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:3.0:1788572600.946844:0:167257:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb74f 00000080:00010000:3.0:1788572600.946848:0:167257:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572600.946850:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8fb58 00010000:00010000:3.0:1788572600.946852:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946858:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:3.0:1788572600.946875:0:167257:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572600.946884:0:167257:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80d.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=13 00000080:00200000:3.0:1788572600.946887:0:167257:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80d.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=getattr 00000002:00010000:3.0:1788572600.946899:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 01 00010000:00010000:3.0:1788572600.946916:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946921:0:167257:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.946926:0:167257:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:3.0:1788572600.946936:0:167257:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610496:192.168.202.157@tcp:101:lfs.0 00000100:00100000:3.0:1788572600.946969:0:167257:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572600.948184:0:167257:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610496:192.168.202.157@tcp:101:lfs.0 00010000:00010000:3.0:1788572600.948195:0:167257:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948202:0:167257:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948207:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948212:0:167257:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948218:0:167257:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948221:0:167257:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a46ec7a00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:3.0:1788572600.948225:0:167257:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948229:0:167257:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948233:0:167257:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948238:0:167257:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:3.0:1788572600.948242:0:167257:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:3.0:1788572600.948246:0:167257:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec7a00/0x8364c713e25bb756 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167257 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:3.0:1788572600.948263:0:167257:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a529b0380 x1875444569610496/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/584 e 0 to 0 dl 1788572616 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:3.0:1788572600.948271:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00010000:00010000:3.0:1788572600.948286:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572600.948294:0:167257:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00010000:3.0:1788572600.948300:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8f930 00010000:00010000:3.0:1788572600.948302:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572600.948307:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572600.948360:0:167257:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572600.948369:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572600.948375:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572600.948381:0:167257:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:3.0:1788572600.948385:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572600.948389:0:167257:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572600.948392:0:167257:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb74f 00000080:00010000:3.0:1788572600.948395:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8fbe8 00010000:00010000:3.0:1788572600.948397:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:3.0:1788572600.948403:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572600.948409:0:167257:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:3.0:1788572600.948413:0:167257:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80d.replay-single, flags=16392 00000080:00200008:3.0:1788572600.948434:0:167257:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572600.948440:0:167257:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572600.948449:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572600.948456:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572600.948464:0:167257:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:3.0:1788572600.948470:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572600.948474:0:167257:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572600.948476:0:167257:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb74f 00000080:00010000:3.0:1788572600.948480:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8fcb8 00010000:00010000:3.0:1788572600.948483:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:3.0:1788572600.948490:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200008:3.0:1788572600.949542:0:167257:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:3.0:1788572600.949547:0:167257:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572600.949555:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572600.949562:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572600.949569:0:167257:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:3.0:1788572600.949574:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572600.949578:0:167257:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572600.949580:0:167257:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb74f 00000080:00010000:3.0:1788572600.949600:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8fcb8 00010000:00010000:3.0:1788572600.949603:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:3.0:1788572600.949608:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572600.949690:0:167257:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:3.0:1788572600.949699:0:167257:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:3.0:1788572600.949705:0:167257:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572600.949715:0:167257:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:3.0:1788572600.949722:0:167257:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:3.0:1788572600.949726:0:167257:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:3.0:1788572600.949728:0:167257:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb74f 00000080:00010000:3.0:1788572600.949731:0:167257:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb74f from it ffffaf7300c8fbb0 00010000:00010000:3.0:1788572600.949733:0:167257:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:3.0:1788572600.949739:0:167257:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000080:00200000:3.0:1788572600.949745:0:167257:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 44 00000080:00200008:3.0:1788572600.949751:0:167257:0:(file.c:1090:ll_file_open()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00000080:00200080:3.0:1788572600.949763:0:167257:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) cmd=c03066f0 arg=7ffc8a72a680 00000080:00200000:3.0:1788572600.949771:0:167257:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08) name=d80d.replay-single stripe_offset=0 stripe_count=1, hash_type=0 00010000:00010000:3.0:1788572600.949787:0:167257:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572600.949831:0:167257:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00010000:00010000:3.0:1788572600.949837:0:167257:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:3.0:1788572600.949840:0:167257:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a46ec6200/0x8364c713e25bb74f lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c0478 expref: -99 pid: 167257 timeout: 0 lvb_type: 0 lru_score: 6 lru_type: 0 00000020:00100000:3.0:1788572600.949860:0:167257:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:3.0:1788572600.949868:0:167257:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a529b0380 x1875444569610752/t0(0) o36->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 568/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572600.949887:0:167257:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610752:192.168.202.157@tcp:36:lfs.0 00000100:00100000:3.0:1788572600.949939:0:167257:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a45990840 going to sleep for 11 seconds 00000100:00100000:3.0:1788572600.951474:0:167257:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167257:1875444569610752:192.168.202.157@tcp:36:lfs.0 00000080:00200000:3.0:1788572600.951495:0:167257:0:(llite_lib.c:3356:ll_read_inode2()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148) 00000080:00200000:3.0:1788572600.951499:0:167257:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0xa:0x0], updating i_size 4096 00000080:00200008:3.0:1788572600.951513:0:167257:0:(file.c:479:ll_file_release()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), flags 300000 00000080:00200000:1.0:1788572601.251402:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572601.251433:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.251485:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 2 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.251496:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.251506:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.251531:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569610880:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.251801:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.253527:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569610880:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.253541:0:167275:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253548:0:167275:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 4/1,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253556:0:167275:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253562:0:167275:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253569:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253574:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 5/2,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.253581:0:167275:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 4/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.253981:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48351f80 x1875444569610880/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572617 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572601.253994:0:167275:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.254006:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8589934599/1 (0 0) 00000080:00200000:1.0:1788572601.254010:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572601.254061:0:167275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200000007:0x1:0x0], updating i_size 69632 00000080:00010000:1.0:1788572601.254065:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb75d 00000080:00010000:1.0:1788572601.254069:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572601.254071:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb75d from it ffffaf7301c77b58 00010000:00010000:1.0:1788572601.254073:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.254079:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572601.254095:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.254110:0:167275:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=d80d.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), flags=29 00000080:00200000:1.0:1788572601.254113:0:167275:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=d80d.replay-single, dir=[0x200000007:0x1:0x0](ffff920a50d46c08), intent=0 00000002:00010000:1.0:1788572601.254126:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: d80d.replay-single,[0x0:0x0:0x0]) in obj [0x200000007:0x1:0x0], intent: lookup flags 0 00010000:00010000:1.0:1788572601.254153:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.254158:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.254163:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.254172:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611008:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.254207:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.255469:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611008:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.255484:0:167275:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255501:0:167275:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x1/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255506:0:167275:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255511:0:167275:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255516:0:167275:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255523:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255528:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255532:0:167275:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.255538:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x10 disposition=0xb, status=0 req@ffff920a48351f80 x1875444569611008/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 592/688 e 0 to 0 dl 1788572617 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572601.255544:0:167275:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255550:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/10 (0 0) 00000080:00200000:1.0:1788572601.255553:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 16 on i1 [0x200000007:0x1:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1788572601.255567:0:167275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0xa:0x0], updating i_size 4096 00000080:00200000:1.0:1788572601.255570:0:167275:0:(namei.c:147:ll_iget()) got inode: [0x200027501:0xa:0x0](ffff920a50d39148): rc = 0 00000080:00010000:1.0:1788572601.255572:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0xa:0x0](ffff920a50d39148) for lock 0x8364c713e25bb764 00000080:00010000:1.0:1788572601.255579:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0xa:0x0](ffff920a50d39148) 00000080:00010000:1.0:1788572601.255580:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb764 from it ffffaf7301c77a28 00010000:00010000:1.0:1788572601.255582:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255612:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572601.255632:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.255640:0:167275:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0xa:0x0](ffff920a50d39148), flags=13 00000080:00200000:1.0:1788572601.255643:0:167275:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0xa:0x0](ffff920a50d39148), intent=getattr 00000002:00010000:1.0:1788572601.255650:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0xa:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572601.255667:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255679:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.255684:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.255694:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611136:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.255724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.256805:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611136:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.256812:0:167275:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256819:0:167275:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256823:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256826:0:167275:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 3/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256831:0:167275:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256834:0:167275:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a78a00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572601.256837:0:167275:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256841:0:167275:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256845:0:167275:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 5/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256849:0:167275:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572601.256852:0:167275:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 3/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.256859:0:167275:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78a00/0x8364c713e25bb76b lrc: 0/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.256866:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a48351f80 x1875444569611136/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572617 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572601.256873:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0xa:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572601.256882:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/10 (0 0) 00000080:00200000:1.0:1788572601.256936:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572601.256946:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.256954:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.256963:0:167275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1788572601.256970:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572601.256974:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572601.256978:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb75d 00000080:00010000:1.0:1788572601.256982:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb75d from it ffffaf7301c77be8 00010000:00010000:1.0:1788572601.256985:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.256993:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572601.257003:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.257009:0:167275:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80d.replay-single, flags=16408 00000080:00200000:1.0:1788572601.257013:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.257022:0:167275:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=remote_dir, dir=[0x200027501:0xa:0x0](ffff920a50d39148), flags=16392 00000080:00200000:1.0:1788572601.257025:0:167275:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=remote_dir, dir=[0x200027501:0xa:0x0](ffff920a50d39148), intent=getattr 00000002:00010000:1.0:1788572601.257033:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: remote_dir,[0x0:0x0:0x0]) in obj [0x200027501:0xa:0x0], intent: getattr flags 01 00010000:00010000:1.0:1788572601.257054:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.257061:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.257067:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 3/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.257081:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611264:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.257114:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.258106:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611264:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.258117:0:167275:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258125:0:167275:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258131:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/1,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258135:0:167275:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 3/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258139:0:167275:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258149:0:167275:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79000 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572601.258152:0:167275:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258158:0:167275:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 4/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258164:0:167275:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 5/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x20e400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258170:0:167275:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572601.258174:0:167275:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 3/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258180:0:167275:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79000/0x8364c713e25bb772 lrc: 0/0,0 mode: --/CR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x2 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.258192:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0x7, status=0 req@ffff920a48351f80 x1875444569611264/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 584/584 e 0 to 0 dl 1788572617 ref 1 fl Complete:RQU/600/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572601.258202:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200027501:0xa:0x0] suppgids 0 -1: rc 0 00010000:00010000:1.0:1788572601.258214:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/10 (0 0) 00000080:00200000:1.0:1788572601.258261:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572601.258276:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.258283:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572601.258292:0:167275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1788572601.258303:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572601.258308:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572601.258311:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb75d 00000080:00010000:1.0:1788572601.258315:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb75d from it ffffaf7301c77b58 00010000:00010000:1.0:1788572601.258318:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572601.258324:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572601.258331:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.258342:0:167275:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80d.replay-single, flags=13 00000080:00200008:1.0:1788572601.258349:0:167275:0:(file.c:6296:ll_getattr_dentry()) START file d80d.replay-single:[0x200027501:0xa:0x0](ffff920a50d39148), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572601.258354:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148),name=d80d.replay-single 00000002:00010000:1.0:1788572601.258363:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0xa:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.258368:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 30 res 8590095617/10 (0 0) 00010000:00010000:1.0:1788572601.258387:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258393:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.258398:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.258416:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611392:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.258455:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.259262:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611392:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.259274:0:167275:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259284:0:167275:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259292:0:167275:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259299:0:167275:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259306:0:167275:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259320:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259325:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 5/2,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259332:0:167275:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 4/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.259342:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff920a48351f80 x1875444569611392/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 576/688 e 0 to 0 dl 1788572617 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1788572601.259352:0:167275:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259361:0:167275:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff920a52412c00 type 13 mode 4 res 8590095617/10 (0 0) 00000080:00200000:1.0:1788572601.259366:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572601.259382:0:167275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0xa:0x0], updating i_size 4096 00000080:00010000:1.0:1788572601.259386:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200027501:0xa:0x0](ffff920a50d39148) for lock 0x8364c713e25bb779 00000080:00010000:1.0:1788572601.259390:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200027501:0xa:0x0](ffff920a50d39148) 00000080:00010000:1.0:1788572601.259392:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb779 from it ffffaf7301c77cb8 00010000:00010000:1.0:1788572601.259394:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/1,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.259399:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00000080:00200008:1.0:1788572601.259894:0:167275:0:(file.c:6296:ll_getattr_dentry()) START file /:[0x200000007:0x1:0x0](ffff920a50d46c08), request_mask 0, flags 0, foreign 0 00000080:00200000:1.0:1788572601.259898:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572601.259907:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.259912:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572601.259919:0:167275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1788572601.259923:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572601.259927:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572601.259928:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb75d 00000080:00010000:1.0:1788572601.259931:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb75d from it ffffaf7301c77cb8 00010000:00010000:1.0:1788572601.259933:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1788572601.259938:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572601.260041:0:167275:0:(file.c:6191:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08),name=/ 00000002:00010000:1.0:1788572601.260048:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1788572601.260052:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572601.260058:0:167275:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00000080:00200000:1.0:1788572601.260062:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1788572601.260064:0:167275:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) 00000080:00010000:1.0:1788572601.260066:0:167275:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff920a50d46c08) for lock 0x8364c713e25bb75d 00000080:00010000:1.0:1788572601.260068:0:167275:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x8364c713e25bb75d from it ffffaf7301c77bf8 00010000:00010000:1.0:1788572601.260070:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1788572601.260075:0:167275:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00000080:00200000:1.0:1788572601.260080:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff920a50d46c08), inode mode 41ed mask 1 00000080:00200000:1.0:1788572601.260084:0:167275:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=d80d.replay-single, flags=267 00000080:00200000:1.0:1788572601.260086:0:167275:0:(file.c:6672:ll_inode_permission()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148), inode mode 41ed mask 44 00000080:00200008:1.0:1788572601.260090:0:167275:0:(file.c:1090:ll_file_open()) START file d80d.replay-single:[0x200027501:0xa:0x0](ffff920a50d39148), flags 300000 00000002:00010000:1.0:1788572601.260098:0:167275:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200027501:0xa:0x0]) in obj [0x200000007:0x1:0x0], intent: open flags 0240300001 00010000:00010000:1.0:1788572601.260119:0:167275:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.260123:0:167275:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000020:00100000:1.0:1788572601.260129:0:167275:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 101, max 7 00000100:00100000:1.0:1788572601.260134:0:167275:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48351f80 x1875444569611648/t0(0) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/131856 e 0 to 0 dl 0 ref 1 fl New:PQU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00010000:00010000:1.0:1788572601.260139:0:167275:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 3/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1788572601.260149:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611648:192.168.202.157@tcp:101:lfs.0 00000100:00100000:1.0:1788572601.260265:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:1.0:1788572601.261328:0:167275:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351f80 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611648:192.168.202.157@tcp:101:lfs.0 00010000:00010000:1.0:1788572601.261346:0:167275:0:(ldlm_request.c:688:ldlm_cli_enqueue_fini()) ### client-side enqueue END (ABORTED) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x0 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261355:0:167275:0:(ldlm_request.c:591:failed_lock_cleanup()) ### setting FL_LOCAL_ONLY | LDLM_FL_FAILED | LDLM_FL_ATOMIC_CB | LDLM_FL_CBPENDING ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261362:0:167275:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/1,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261378:0:167275:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x10206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261387:0:167275:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261397:0:167275:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79c00 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572601.261401:0:167275:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261408:0:167275:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 4/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x206400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261415:0:167275:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 5/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x20e400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261421:0:167275:0:(ldlm_request.c:1407:ldlm_cli_cancel_local()) not sending request (at caller's instruction) 00010000:00010000:1.0:1788572601.261426:0:167275:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 3/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1788572601.261433:0:167275:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79c00/0x8364c713e25bb780 lrc: 0/0,0 mode: --/CR res: [0x200000007:0x1:0x0].0x0 bits 0x1/0x1 rrc: 3 type: IBT flags: 0x4a0f400000000 pid: 167275 initiator: MDT0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1788572601.261446:0:167275:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x1 disposition=0x2b, status=0 req@ffff920a48351f80 x1875444569611648/t352187318286(352187318286) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572617 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000002:00100000:1.0:1788572601.261458:0:167275:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351f80 x1875444569611648/t352187318286(352187318286) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 648/664 e 0 to 0 dl 1788572617 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200000:1.0:1788572601.261469:0:167275:0:(namei.c:952:ll_intent_lock()) intent lock 1 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 0 00000080:00200000:1.0:1788572601.261484:0:167275:0:(llite_lib.c:3119:ll_update_inode()) inode=[0x200027501:0xa:0x0], updating i_size 4096 00000002:00100000:1.0:1788572601.261496:0:167275:0:(mdc_request.c:811:mdc_set_open_replay_data()) @@@ Set up open replay data req@ffff920a48351f80 x1875444569611648/t352187318286(352187318286) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572617 ref 2 fl Complete:RPQU/604/0 rc 301/301 job:'lfs.0' uid:0 gid:0 projid:0 00000080:00200080:1.0:1788572601.261546:0:167275:0:(dir.c:2004:ll_dir_ioctl()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148) cmd=c03066f0 arg=7fff12c19e60 00000080:00200000:1.0:1788572601.261558:0:167275:0:(dir.c:451:ll_dir_setdirstripe()) VFS Op:inode=[0x200027501:0xa:0x0](ffff920a50d39148) name=remote_dir stripe_offset=1 stripe_count=0, hash_type=0 00000020:00100000:1.0:1788572601.261618:0:167275:0:(genops.c:2428:obd_get_mod_rpc_slot()) lustre-MDT0001-mdc-ffff920a47b93800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1788572601.261625:0:167275:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff920a48353100 x1875444569611904/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 560/576 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572601.261638:0:167275:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353100 pname:cluuid:pid:xid:nid:opc:job lfs:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:167275:1875444569611904:192.168.202.157@tcp:36:lfs.0 00000100:00100000:1.0:1788572601.261673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 11 seconds 00000100:00100000:2.0:1788572601.263756:0:2391:0:(events.c:344:request_in_callback()) peer: 12345-192.168.202.157@tcp (source: 12345-192.168.202.157@tcp) 00000100:00100000:3.0:1788572601.263801:0:160922:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1875444584520832 00000100:00100000:3.0:1788572601.263808:0:160922:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1875444584520832 00000100:00100000:3.0:1788572601.263847:0:160922:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.202.157@tcp, seq: 12 00000100:00100000:3.0:1788572601.263855:0:160922:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff920a60ac7b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584520832:12345-192.168.202.157@tcp:104: 00010000:00010000:3.0:1788572601.263866:0:160922:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:3.0:1788572601.263912:0:160922:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:3.0:1788572601.263918:0:160922:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff920a60ac7b80 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb01_001:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b+5:188053:x1875444584520832:12345-192.168.202.157@tcp:104: Request processed in 64us (166us total) trans 0 rc 0/0 00000100:00100000:3.0:1788572601.263926:0:160922:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.202.157@tcp, seq: 12 00010000:00010000:1.0:1788572601.263934:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1788572601.263942:0:160930:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x400000000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.263952:0:160930:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff920a60a79400 already unused, calling callback (ffffffffc1ae7630) 00010000:00010000:1.0:1788572601.263957:0:160930:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.263966:0:160930:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.263974:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 3/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.263991:0:160930:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788572601.264016:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 2/0,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.264021:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1788572601.264046:0:160930:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 1/0,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.264059:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79400/0x8364c713e25bb779 lrc: 0/0,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x13/0x13 rrc: 3 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0x265cfe11109c0504 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1788572601.264070:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572601.264073:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572601.264075:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572601.264081:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612032:192.168.202.157@tcp:103:ldlm_bl.0 00000100:00100000:0.0:1788572601.264924:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612032:192.168.202.157@tcp:103:ldlm_bl.0 00010000:00010000:3.0:1788572601.755698:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572602.331796:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569612160:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.331811:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612288:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.331902:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612544:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572602.331905:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572602.331951:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612672:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572602.333271:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569612160:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.333289:0:2398:0:(client.c:3126:ptlrpc_free_committed()) lustre-MDT0000-mdc-ffff920a47b93800: committing for last_committed 352187318286 gen 1 00000100:00100000:1.0:1788572602.333302:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612672:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.333312:0:2398:0:(client.c:3175:ptlrpc_free_committed()) @@@ keeping (FL_REPLAY) req@ffff920a48351f80 x1875444569611648/t352187318286(352187318286) o101->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/664 e 0 to 0 dl 1788572617 ref 1 fl Complete:RPQU/604/ffffffff rc 301/-1 job:'lfs.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1788572602.333324:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612416:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.333325:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612288:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572602.333346:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569612544:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572602.779738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572603.803726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572604.827755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572605.851689:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572606.875747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572607.451985:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569612800:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572607.452017:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572607.452086:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569613184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572607.452097:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572607.452140:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613312:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572607.899670:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572608.923816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572609.947673:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788572610.971747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:3.0:1788572611.995698:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572612.571738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 1 seconds 00000100:00100000:0.0:1788572612.571799:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569613440:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.571832:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613568:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.571899:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613824:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572612.571900:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569613696:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572612.571944:0:2398:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1788572612.571954:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613952:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572613.019760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572613.595673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 1 seconds 00010000:00010000:3.0:1788572614.043708:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572614.619690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 1 seconds 00010000:00010000:1.0:1788572615.067834:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572615.643674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 1 seconds 00010000:00010000:3.0:1788572616.091719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572616.667691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 1 seconds 00010000:00010000:1.0:1788572617.115738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:3.0:1788572617.691705:0:167275:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572601/real 1788572601] req@ffff920a48353100 x1875444569611904/t0(0) o36->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 560/576 e 0 to 1 dl 1788572617 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'lfs.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572617.691730:0:167275:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:3.0:1788572617.691735:0:167275:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0001-mdc-ffff920a47b93800: Connection to lustre-MDT0001 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:0.0:1788572617.691740:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569614208:192.168.202.157@tcp:400:kworker.0 00000100:00080000:3.0:1788572617.691742:0:167275:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0001_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1788572617.691760:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a479a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569614080:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572617.691760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572617.691784:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572617.691791:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572617.691795:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572617.691802:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9340 00000100:00080000:2.0:1788572617.691814:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572617.691822:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572617.691823:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:0:1875444569614336:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572617.691835:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569614592:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572617.691836:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569614464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572617.691842:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569614720/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572617.691874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569614720:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572618.139711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572618.715682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572619.163766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572619.739685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572620.187704:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572620.763772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572621.211766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572621.787753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572622.235718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572622.811731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572622.811844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572622.811849:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572622.811884:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569614848:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572622.811889:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569614976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572622.812037:0:2398:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569615232:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572622.812050:0:2397:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569615104:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572623.259799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572623.835669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00000400:1.0:1788572623.835679:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff920a48353800 x1875444569613056/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572623.835714:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48353800 x1875444569613056/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572623.835722:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a51ac4800 already not connected (conn 15, was 14): CONNECTING 00000100:00000400:0.0:1788572623.835726:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff920a48350a80 x1875444569612928/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572623.835731:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff920a48351500 x1875444569613312/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572623.835755:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48350a80 x1875444569612928/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572623.835761:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48351500 x1875444569613312/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572623.835767:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:00080000:1.0:1788572623.835771:0:2398:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788572623.835774:0:2397:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff920a47b93800: Connection to lustre-MDT0000 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:02000400:1.0:1788572623.835778:0:2398:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff920a47b93800: Connection to lustre-OST0001 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:0.0:1788572623.835783:0:2397:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00080000:1.0:1788572623.835785:0:2398:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00000400:0.0:1788572623.835811:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff920a48351c00 x1875444569613184/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572623.835815:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572607/real 1788572607] req@ffff920a48350700 x1875444569612800/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572623.835828:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48350700 x1875444569612800/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572623.835834:0:2398:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from FULL to DISCONN 00000100:00100000:0.0:1788572623.835837:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48351c00 x1875444569613184/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:02020000:1.0:1788572623.835841:0:2398:0:(import.c:186:ptlrpc_set_import_discon()) MGC192.168.202.157@tcp: Connection to MGS (at 192.168.202.157@tcp) was lost; in progress operations using this service will fail 00000100:00080000:1.0:1788572623.835848:0:2398:0:(import.c:192:ptlrpc_set_import_discon()) import MGS@192.168.202.157@tcp for MGC192.168.202.157@tcp not replayable, auto-deactivating 00000100:00080000:1.0:1788572623.835852:0:2398:0:(import.c:138:ptlrpc_deactivate_import_nolock()) setting import MGS INVALID 00000100:00080000:0.0:1788572623.835853:0:2397:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:0.0:1788572623.835860:0:2397:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff920a47b93800: Connection to lustre-OST0000 (at 192.168.202.157@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00100000:1.0:1788572623.835860:0:2398:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a48350700 x1875444569612800/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 1 dl 1788572623 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572623.835863:0:2397:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:1.0:1788572623.835872:0:2398:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a48353b80 x1875444569613440/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572628 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572623.835879:0:2398:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a479a3b80 x1875444569614080/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572633 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572623.835887:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569613440:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.835889:0:2398:0:(client.c:3591:ptlrpc_abort_inflight()) @@@ inflight req@ffff920a52892300 x1875444569614848/t0(0) o400->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 224/224 e 0 to 0 dl 1788572638 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 10000000:01000000:1.0:1788572623.835897:0:2398:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808001 00000100:00080000:1.0:1788572623.835900:0:2398:0:(pinger.c:193:ptlrpc_pinger_ir_down()) IR down 10000000:01000000:1.0:1788572623.835901:0:2398:0:(mgc_request.c:1168:mgc_import_event()) import event 0x808002 00000100:00080000:1.0:1788572623.835903:0:2398:0:(import.c:418:ptlrpc_pinger_force()) MGS: waking up pinger s:DISCONN 00000100:00100000:0.0:1788572623.835912:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569612928:192.168.202.157@tcp:400:kworker.0 00000100:00100000:0.0:1788572623.835921:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569613184:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.835928:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569614080:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.835937:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613056:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.835943:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613312:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572623.835951:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:f79285ed-4685-45a8-8731-5e2522a143f3:2398:1875444569612800:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572623.835952:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.835971:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572623.835974:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572623.835976:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:f79285ed-4685-45a8-8731-5e2522a143f3:2397:1875444569614848:192.168.202.157@tcp:400:kworker.0 00000100:00100000:3.0:1788572623.835984:0:2399:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:f79285ed-4685-45a8-8731-5e2522a143f3:0:1875444569615360:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572623.835986:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9287 00000100:00080000:2.0:1788572623.836011:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572623.836026:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572623.836045:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569615488/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572623.836067:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.836072:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572623.836075:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.836079:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572623.836082:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572623.836088:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9349 00000100:00080000:2.0:1788572623.836096:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572623.836099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615488:192.168.202.157@tcp:38:kworker.0 00000008:00100000:2.0:1788572623.836107:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572623.836124:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569615616/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572623.836136:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.836139:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572623.836142:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572623.836146:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9349 00000100:00080000:2.0:1788572623.836152:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572623.836157:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572623.836169:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569615744/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572623.836184:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.836187:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572623.836189:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572623.836193:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9283 00000100:00080000:2.0:1788572623.836200:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572623.836205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615616:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572623.836217:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569615872/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572623.836232:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572623.836236:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572623.836241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615744:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572623.836271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569615872:192.168.202.157@tcp:250:kworker.0 00010000:00010000:3.0:1788572624.283712:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572624.859727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572625.307803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572625.883770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572626.331715:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572626.907708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572627.355728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572627.931697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572627.931741:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572617/real 1788572617] req@ffff920a60ac4e00 x1875444569614720/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572627.931785:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569614720/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572627 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572627.931802:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572627.931826:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572627.931845:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569614720:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572627.931865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.931872:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.931875:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.931880:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.931883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.931886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572627.931889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572627.931897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9376 00000100:00080000:3.0:1788572627.931906:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:3.0:1788572627.931914:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572627.931926:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572627.931956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569616000/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572627.931983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.931993:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572627.931997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572627.932002:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572627.932039:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616000:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572628.379719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:0.0:1788572628.955726:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff920a48353480 x1875444569613696/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572628.955754:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48353480 x1875444569613696/t0(0) o400->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572628.955763:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff920a479a3800 x1875444569613952/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572628.955764:0:2397:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a51ac4800 already not connected (conn 16, was 14): CONNECTING 00000100:00100000:0.0:1788572628.955782:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569613696:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572628.955789:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a479a3800 x1875444569613952/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572628.955809:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1000 already not connected (conn 6, was 5): CONNECTING 00000100:00000400:1.0:1788572628.955817:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff920a48350380 x1875444569613568/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.955833:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a48350380 x1875444569613568/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572628.955833:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572628.955843:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a4549f800 already not connected (conn 17, was 16): CONNECTING 00000100:00000400:1.0:1788572628.955848:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572612/real 1788572612] req@ffff920a479a0a80 x1875444569613824/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572628.955862:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a479a0a80 x1875444569613824/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572628 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572628.955866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572628.955867:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1800 already not connected (conn 10, was 9): CONNECTING 00000100:00080000:2.0:1788572628.955876:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572628.955881:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572628.955882:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a3800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613952:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572628.955891:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572628.955896:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572628.955900:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572628.955902:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613568:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572628.955904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572628.955909:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572628.955913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572628.955916:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572628.955919:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a0a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569613824:192.168.202.157@tcp:400:kworker.0 00010000:00010000:1.0:1788572629.403716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572629.979763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572630.427790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572631.003823:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572631.451793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572632.027752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572632.475780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572632.987704:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.987715:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.987719:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.987723:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.987726:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.987729:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.987732:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.987736:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572632.987738:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572632.987742:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:0.0:1788572633.051778:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572617/real 1788572617] req@ffff920a52892d80 x1875444569614592/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572633.051779:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572617/real 1788572617] req@ffff920a52893100 x1875444569614464/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572633.051800:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788572633.051808:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572623/real 1788572623] req@ffff920a60ac4700 x1875444569615872/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572633.051839:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52892d80 x1875444569614592/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572633.051872:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1000 already not connected (conn 6, was 5): CONNECTING 00000100:00100000:2.0:1788572633.051874:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569615872/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572633.051887:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572623/real 1788572623] req@ffff920a60ac4380 x1875444569615616/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572633.051895:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569614592:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572633.051897:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52893100 x1875444569614464/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572633.051910:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569615616/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572633.051928:0:2397:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1800 already not connected (conn 10, was 9): CONNECTING 00000100:00100000:2.0:1788572633.051935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572623/real 1788572623] req@ffff920a60ac4000 x1875444569615744/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1788572633.051947:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572617/real 1788572617] req@ffff920a479a1f80 x1875444569614208/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572633.051954:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569615744/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572633.051992:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a479a1f80 x1875444569614208/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572633.052020:0:2397:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a4549f800 already not connected (conn 17, was 16): CONNECTING 00000100:00100000:2.0:1788572633.052043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572623/real 1788572623] req@ffff920a60ac4a80 x1875444569615488/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572633.052053:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569614464:192.168.202.157@tcp:400:kworker.0 00000100:00100000:2.0:1788572633.052076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569615488/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572633 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572633.052077:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a479a1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569614208:192.168.202.157@tcp:400:kworker.0 00000100:00080000:2.0:1788572633.052111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572633.052130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572633.052137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569615872:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572633.052144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572633.052150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572633.052154:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615616:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788572633.052160:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572633.052160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572633.052166:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572633.052168:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572633.052171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572633.052175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615744:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788572633.052176:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9382 00000100:00080000:2.0:1788572633.052180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572633.052185:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572633.052188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569615488:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572633.052189:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572633.052197:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572633.052235:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a51a2f480 x1875444569616128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572633.052261:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572633.052265:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572633.052267:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572633.052271:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9382 00000100:00080000:0.0:1788572633.052275:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572633.052279:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572633.052291:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572633.052291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569616128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572633.052307:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569616256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572633.052317:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572633.052320:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572633.052323:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572633.052326:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572633.052327:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572633.052330:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9382 00000100:00080000:0.0:1788572633.052333:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572633.052337:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572633.052342:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572633.052367:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569616384/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572633.052377:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572633.052379:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572633.052381:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572633.052384:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9382 00000100:00080000:0.0:1788572633.052388:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 5s 00000100:00080000:0.0:1788572633.052392:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572633.052396:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616256:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788572633.052397:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572633.052409:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569616512/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572633.052425:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572633.052428:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616384:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788572633.052429:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572633.052461:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616512:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572633.499780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572634.075853:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572634.523725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572635.099719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572635.547764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572636.123671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572636.571732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572637.147672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572637.595750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572638.171715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572638.171722:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572638.171733:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572638.171739:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572638.171744:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572638.171748:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572638.171752:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572638.171756:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572638.171760:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572638.171766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572638.171771:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00000400:1.0:1788572638.171819:0:2397:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff920a52891880 x1875444569615104/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572638.171851:0:2397:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52891880 x1875444569615104/t0(0) o400->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572638.171858:0:2397:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1800 already not connected (conn 11, was 9): CONNECTING 00000100:00100000:1.0:1788572638.171873:0:2397:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2397:1875444569615104:192.168.202.157@tcp:400:kworker.0 00000100:00000400:1.0:1788572638.171900:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff920a52891180 x1875444569614976/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572638.171910:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52891180 x1875444569614976/t0(0) o400->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572638.171915:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff920a4549f800 already not connected (conn 18, was 16): CONNECTING 00000100:00000400:1.0:1788572638.171919:0:2398:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572622/real 1788572622] req@ffff920a52890e00 x1875444569615232/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572638.171926:0:2398:0:(client.c:2548:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff920a52890e00 x1875444569615232/t0(0) o400->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 224/224 e 0 to 1 dl 1788572638 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572638.171935:0:2398:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff920a51ac1000 already not connected (conn 7, was 5): CONNECTING 00000100:00100000:1.0:1788572638.171939:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569614976:192.168.202.157@tcp:400:kworker.0 00000100:00100000:1.0:1788572638.171942:0:2398:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2398:1875444569615232:192.168.202.157@tcp:400:kworker.0 00010000:00010000:3.0:1788572638.619714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572639.195811:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572639.643808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572640.219737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572640.667751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572641.243787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572641.691780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572642.267685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572642.267765:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572627/real 1788572627] req@ffff920a529b3b80 x1875444569616000/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572642 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572642.267793:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569616000/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572642 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572642.267812:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572642.267832:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572642.267835:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616000:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572642.267865:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572642.267874:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572642.267877:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572642.267879:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572642.267882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572642.267884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572642.267888:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572642.267897:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9386 00000100:00080000:1.0:1788572642.267909:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572642.267916:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572642.267929:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572642.267973:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569616640/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572642.268004:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572642.268009:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572642.268012:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572642.268017:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572642.268046:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616640:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572642.715707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572643.291739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572643.739707:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572644.315742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572644.763686:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572645.339713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572645.787812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572646.363738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572646.811731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572647.387730:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572647.387738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572647.387742:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.387749:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.387753:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.387757:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.387762:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.387766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.387771:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:0.0:1788572647.387774:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572647.387777:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788572647.835779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572648.411673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788572648.411724:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572633/real 1788572633] req@ffff920a48350700 x1875444569616512/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411747:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569616512/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411755:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572633/real 1788572633] req@ffff920a48350380 x1875444569616256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411763:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569616256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411769:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572633/real 1788572633] req@ffff920a48353480 x1875444569616384/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411778:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569616384/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411784:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572633/real 1788572633] req@ffff920a51a2f480 x1875444569616128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572648.411792:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a51a2f480 x1875444569616128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572648 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572648.411809:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572648.411828:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572648.411832:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616512:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788572648.411837:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572648.411846:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572648.411851:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616256:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572648.411856:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572648.411860:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.411865:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572648.411867:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.411870:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572648.411873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572648.411878:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788572648.411882:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9391 00000100:00100000:0.0:1788572648.411890:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616384:192.168.202.157@tcp:8:kworker.0 00000100:00080000:1.0:1788572648.411892:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:0.0:1788572648.411902:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572648.411903:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788572648.411915:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:1.0:1788572648.411917:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572648.411921:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a51a2f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569616128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572648.411950:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569616768/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572648.411972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.411977:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1788572648.411981:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.411984:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572648.411985:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572648.411990:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9391 00000100:00100000:0.0:1788572648.411993:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616768:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572648.411994:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572648.412002:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572648.412007:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572648.412019:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569616896/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572648.412027:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.412029:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572648.412030:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572648.412033:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9391 00000100:00080000:1.0:1788572648.412046:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572648.412048:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572648.412052:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572648.412059:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569617024/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572648.412069:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.412072:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572648.412073:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572648.412076:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9391 00000100:00080000:1.0:1788572648.412079:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 10s 00000100:00080000:1.0:1788572648.412081:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572648.412088:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569617152/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572648.412098:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572648.412103:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:0.0:1788572648.412128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616896:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572648.412160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617024:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572648.412228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569617152:192.168.202.157@tcp:250:kworker.0 00010000:00010000:3.0:1788572648.859726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572649.435686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572649.883787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572650.459727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572650.907766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572651.483683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572651.931794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572652.507731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572652.955745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572653.531706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000800:00000100:0.0:1788572653.531713:0:2388:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.202.157@tcp (192.168.202.157:988) timed out; the network or node may be down. 00000100:00080000:2.0:1788572653.531815:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572653.531825:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572653.531829:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572653.531833:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572653.531836:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572653.531840:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572653.531843:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572653.531847:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1788572653.531851:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572653.531856:0:155040:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1788572653.979773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572654.555701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572655.003717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572655.579704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572656.027746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572656.603733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572657.051713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572657.627698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572658.075782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572658.651771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572658.651804:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572658.651813:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572658.651817:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572658.651821:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572658.651825:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572658.651829:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572658.651833:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572658.651836:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572658.651839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572658.651843:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:3.0:1788572659.099729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572659.675738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572660.123783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572660.699687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572661.147677:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572661.723718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572662.171767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572662.747661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788572662.747667:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572642/real 1788572642] req@ffff920a52890e00 x1875444569616640/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572662.747695:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569616640/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572662 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572662.747720:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572662.747742:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572662.747746:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616640:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572662.747793:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572662.747797:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572662.747799:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572662.747801:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572662.747802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572662.747804:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572662.747806:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572662.747811:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9400 00000100:00080000:3.0:1788572662.747819:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572662.747826:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572662.747837:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572662.747862:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569617280/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572662.747885:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572662.747889:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572662.747892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572662.747895:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1788572662.747975:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617280:192.168.202.157@tcp:38:kworker.0 00010000:00010000:3.0:1788572663.195674:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572663.771672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572663.771915:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:1.0:1788572663.771952:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572663.771965:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788572663.772581:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572662/real 1788572663] req@ffff920a529b1500 x1875444569617280/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572687 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572663.772622:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569617280/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572687 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572663.772633:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572663.772637:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572663.772641:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617280:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572664.219758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572664.795759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572665.243728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572665.819694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572666.267795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572666.843711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572667.291812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572667.867694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572667.867700:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572667.867711:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572667.867716:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572667.867721:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572667.867724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572667.867729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572667.867735:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572667.867748:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9421 00000100:00080000:3.0:1788572667.867760:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572667.867771:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572667.867788:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572667.867829:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569617408/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572667.867852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572667.867857:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572667.867860:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572667.867864:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1788572667.867898:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617408:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572668.315771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572668.891815:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572648/real 1788572648] req@ffff920a52891880 x1875444569616896/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891871:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569616896/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572668.891873:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788572668.891889:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572648/real 1788572648] req@ffff920a52893100 x1875444569617024/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891907:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569617024/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891920:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572648/real 1788572648] req@ffff920a52890700 x1875444569617152/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891934:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569617152/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891947:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788572648/real 1788572648] req@ffff920a52891180 x1875444569616768/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572668.891959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569616768/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572668 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572668.891987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572668.892018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572668.892024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616896:192.168.202.157@tcp:8:kworker.0 00000100:00080000:2.0:1788572668.892031:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572668.892037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572668.892041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617024:192.168.202.157@tcp:8:kworker.0 00000100:00080000:2.0:1788572668.892046:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572668.892051:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572668.892054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569617152:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572668.892059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572668.892064:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572668.892068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569616768:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572668.892168:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572668.892175:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572668.892178:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572668.892186:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9406 00000100:00080000:3.0:1788572668.892199:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572668.892208:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572668.892251:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569617536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572668.892277:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572668.892281:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572668.892283:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572668.892287:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9406 00000100:00080000:3.0:1788572668.892296:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572668.892301:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572668.892314:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572668.892327:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569617664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572668.892337:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572668.892342:0:94873:0:(pinger.c:247:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:3.0:1788572668.892345:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572668.892348:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572668.892350:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572668.892354:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9406 00000100:00080000:3.0:1788572668.892358:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572668.892363:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572668.892369:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572668.892380:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569617792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572668.892390:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572668.892393:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572668.892395:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572668.892399:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9406 00000100:00080000:3.0:1788572668.892403:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 15s 00000100:00080000:3.0:1788572668.892407:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572668.892412:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572668.892421:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569617920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572668.892493:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569617536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572668.892628:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572668.892663:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572668.892685:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617920:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572669.339772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572669.915710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572669.915970:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:1.0:1788572669.916023:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572669.916043:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572669.916102:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572669.916119:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572669.916136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572669] req@ffff920a529b0e00 x1875444569617536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1788572669.916138:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572669.916154:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572669.916176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569617536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.916192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.916199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572669.916204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569617536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572669.916214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572669] req@ffff920a529b1f80 x1875444569617664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.916230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569617664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.916239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.916244:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572669.916247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572669.916255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572669] req@ffff920a529b0000 x1875444569617792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.916276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569617792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.916285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.916289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572669.916293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572669.916301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572668/real 1788572669] req@ffff920a529b3100 x1875444569617920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.916328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569617920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.916342:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.916346:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572669.916349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572669.916357:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572667/real 1788572669] req@ffff920a529b2d80 x1875444569617408/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572692 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572669.916370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569617408/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572692 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572669.916379:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572669.916383:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572669.916387:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569617408:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572670.363842:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572670.939725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572671.387744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572671.963796:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572672.411777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572672.923674:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572672.923686:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572672.923691:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572672.923702:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9427 00000100:00080000:3.0:1788572672.923714:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572672.923722:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572672.923766:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569618048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572672.923794:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572672.923797:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572672.923799:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572672.923802:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9427 00000100:00080000:3.0:1788572672.923805:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572672.923809:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572672.923821:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572672.923836:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569618176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572672.923847:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569618048:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572672.923854:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572672.923859:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572672.923862:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572672.923867:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9426 00000100:00080000:3.0:1788572672.923870:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572672.923873:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572672.923879:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572672.923890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569618304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572672.923897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572672.923901:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572672.923903:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572672.923905:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9427 00000100:00080000:3.0:1788572672.923917:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572672.923919:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572672.923923:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572672.923931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569618432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572672.923938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572672.923941:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572672.923942:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572672.923944:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9427 00000100:00080000:3.0:1788572672.923946:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572672.923949:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572672.923952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572672.923959:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569618560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572672.923979:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572672.924010:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572672.924031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572672.924047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572672.987754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572672.987909:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:1.0:1788572672.987953:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572672.987968:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572672.988002:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572672.988015:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572672.988025:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572672/real 1788572672] req@ffff920a529b2a00 x1875444569618176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1788572672.988026:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572672.988039:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572672.988051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569618176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572672.988063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572672.988069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572672.988073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572672.988081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572672/real 1788572672] req@ffff920a529b3480 x1875444569618304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572672.988096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569618304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572672.988105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572672.988110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572672.988113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572672.988123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572672/real 1788572672] req@ffff920a529b1c00 x1875444569618432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572672.988136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569618432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572672.988144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572672.988149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572672.988152:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572672.988160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572672/real 1788572672] req@ffff920a529b2300 x1875444569618560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572672.988171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569618560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572672.988178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572672.988181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572672.988184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572672.988190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572672/real 1788572672] req@ffff920a529b0700 x1875444569618048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572672.988201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569618048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572697 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572672.988208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572672.988211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572672.988214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569618048:192.168.202.157@tcp:250:kworker.0 00010000:00010000:3.0:1788572673.435721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572674.011751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572674.459702:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572675.035727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572675.483692:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572676.059823:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572676.507734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572677.083797:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572677.531718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572678.107683:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572678.107692:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572678.107696:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572678.107704:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9431 00000100:00080000:3.0:1788572678.107713:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572678.107720:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572678.107756:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569618688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572678.107793:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572678.107802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572678.107810:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572678.107815:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572678.107823:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9431 00000100:00100000:0.0:1788572678.107825:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569618688:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572678.107827:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572678.107834:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572678.107843:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572678.107856:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569618816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572678.107865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572678.107868:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572678.107870:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572678.107873:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9431 00000100:00080000:3.0:1788572678.107875:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572678.107878:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572678.107881:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572678.107889:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569618944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572678.107895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572678.107899:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572678.107900:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572678.107903:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9431 00000100:00080000:3.0:1788572678.107906:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572678.107908:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572678.107913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572678.107914:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572678.107923:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569619072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572678.107931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572678.107933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572678.107934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572678.107938:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9431 00000100:00080000:3.0:1788572678.107942:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572678.107945:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572678.107948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572678.107949:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572678.107959:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569619200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572678.107988:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572678.108019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619200:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572678.555722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572679.131842:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788572679.132006:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:3.0:1788572679.133350:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572679.133367:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572679.133416:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572679.133427:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572679.133436:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572679.133445:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572679.133729:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572679] req@ffff920a529b3480 x1875444569619072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572679.133767:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569619072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572679.133779:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572679.133785:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572679.133790:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572679.133798:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572679] req@ffff920a529b2a00 x1875444569619200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572679.133821:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569619200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572679.133831:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572679.133834:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572679.133839:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572679.133847:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572679] req@ffff920a529b2300 x1875444569618816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572679.133860:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569618816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572679.133870:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572679.133874:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572679.133878:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572679.133885:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572679] req@ffff920a529b1c00 x1875444569618944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572679.133898:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569618944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572679.133906:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572679.133910:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572679.133913:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569618944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572679.133920:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572678/real 1788572679] req@ffff920a529b0700 x1875444569618688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572679.133933:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569618688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572679.133941:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572679.133944:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572679.133948:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569618688:192.168.202.157@tcp:250:kworker.0 00010000:00010000:3.0:1788572679.579756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572680.155693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572680.603735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572681.179764:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572681.627700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572682.203663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572682.651772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572683.227664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572683.227664:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.227674:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572683.227678:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.227696:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9436 00000100:00080000:3.0:1788572683.227706:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.227715:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572683.227747:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569619328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.227771:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.227776:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572683.227778:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.227783:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9436 00000100:00080000:3.0:1788572683.227788:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.227793:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572683.227803:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569619328:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788572683.227804:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572683.227822:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569619456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.227832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.227834:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572683.227836:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572683.227839:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9436 00000100:00080000:3.0:1788572683.227851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.227854:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788572683.227859:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788572683.227859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572683.227878:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569619584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572683.227889:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572683.227894:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572683.227896:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572683.227901:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff920a529b2d80 x1875444569619328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572683.227902:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9436 00000100:00080000:3.0:1788572683.227909:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.227918:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572683.227926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572683.227933:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569619328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572683.227940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569619712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572683.227949:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572683.227954:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572683.227958:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788572683.227960:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572683.227964:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572683.227966:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569619328:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572683.227971:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9436 00000100:00080000:3.0:1788572683.227978:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572683.227982:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572683.227987:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572683.227999:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572683.228001:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569619840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572683.228051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572683.228066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff920a529b3100 x1875444569619456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.228079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569619456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.228087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.228090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572683.228094:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572683.228100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572683.228118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572683.228128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff920a529b0000 x1875444569619584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.228153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569619584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.228167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.228170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572683.228173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572683.228178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619712:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572683.228200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572683.228211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff920a529b1f80 x1875444569619712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.228224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569619712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.228232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.228235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572683.228238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572683.228254:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572683.228275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572683.228286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572683/real 1788572683] req@ffff920a529b0e00 x1875444569619840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572683.228298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569619840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572683.228306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572683.228309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572683.228313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569619840:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572683.675746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572684.251686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572684.699755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572685.275664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572685.723686:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572686.299698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572686.747709:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572687.323679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572687.771676:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572688.347733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572688.347875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.347882:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572688.347886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.347897:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9441 00000100:00080000:3.0:1788572688.347907:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.347916:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572688.347952:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569619968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.347980:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.347985:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572688.347988:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.347993:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9441 00000100:00080000:3.0:1788572688.347997:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.348001:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572688.348023:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572688.348039:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569620096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572688.348041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569619968:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572688.348059:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.348064:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572688.348066:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.348072:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9441 00000100:00080000:3.0:1788572688.348076:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.348080:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572688.348086:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572688.348098:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569620224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.348108:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.348113:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572688.348115:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.348119:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9441 00000100:00080000:3.0:1788572688.348122:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572688.348125:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572688.348131:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572688.348142:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569620352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572688.348152:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572688.348156:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572688.348158:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572688.348162:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9441 00000100:00080000:3.0:1788572688.348165:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572688.348171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620096:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572688.348466:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572688.348470:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620224:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788572688.348477:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572688.348498:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569620480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572688.348514:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572688.348549:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572688.795757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572689.371698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572689.819789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572690.395726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572690.843768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572691.419728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572691.420051:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:1.0:1788572691.420108:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572691.420126:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572691.420174:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572691.420187:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572691.420200:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572691.420215:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572691.420239:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572691] req@ffff920a529b3b80 x1875444569620096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572691.420287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569620096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572691.420302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572691.420308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572691.420313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620096:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572691.420323:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572691] req@ffff920a529b0380 x1875444569620224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572691.420338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569620224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572691.420347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572691.420352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572691.420361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572691.420372:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572691] req@ffff920a529b1180 x1875444569620352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572691.420386:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569620352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572691.420395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572691.420400:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572691.420404:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572691.420413:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572691] req@ffff920a529b1500 x1875444569619968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572691.420427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569619968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572691.420435:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572691.420440:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572691.420444:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569619968:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572691.420497:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572688/real 1788572691] req@ffff920a529b0e00 x1875444569620480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572691.420511:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569620480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572691.420520:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572691.420523:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572691.420527:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572691.867717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572692.443691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572692.891746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572693.467760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572693.467785:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.467793:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572693.467797:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.467807:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9446 00000100:00080000:1.0:1788572693.467817:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.467825:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572693.467882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569620608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.467938:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.467944:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572693.467947:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.467953:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9446 00000100:00080000:1.0:1788572693.467961:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.467968:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572693.467979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572693.468000:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569620608:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572693.468006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569620736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.468022:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.468026:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572693.468035:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.468040:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9446 00000100:00080000:1.0:1788572693.468044:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.468050:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572693.468057:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572693.468075:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569620864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.468085:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.468089:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572693.468090:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.468095:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9446 00000100:00080000:1.0:1788572693.468100:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.468105:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788572693.468113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788572693.468114:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572693.468134:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569620992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572693.468147:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572693.468150:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572693.468154:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572693.468161:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9446 00000100:00080000:1.0:1788572693.468165:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572693.468174:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572693.468180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572693] req@ffff920a52890e00 x1875444569620608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572693.468184:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572693.468207:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569621120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572693.468228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569620608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572693.468245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572693.468252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572693.468257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569620608:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572693.468305:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620736:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572693.468341:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572693.468356:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572693] req@ffff920a52893480 x1875444569620736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572693.468369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569620736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572693.468379:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572693.468382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572693.468386:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620736:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572693.468394:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572693.468424:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572693.468436:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572693] req@ffff920a52891f80 x1875444569620864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572693.468450:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569620864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572693.468457:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572693.468461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572693.468464:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572693.468471:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620992:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572693.468493:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572693.468503:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572693] req@ffff920a52892a00 x1875444569620992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572693.469376:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569620992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572693.469391:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572693.469395:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572693.469399:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569620992:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572693.469410:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572693.469464:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572693.469694:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572693/real 1788572693] req@ffff920a52891c00 x1875444569621120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572693.469718:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569621120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572693.469727:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572693.469731:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572693.469735:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621120:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572693.915732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572694.491700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572694.939802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572695.515674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572695.963711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572696.539656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572696.987778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572697.563668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572698.011734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572698.587728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572698.587806:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572698.587812:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572698.587816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572698.587828:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9451 00000100:00080000:1.0:1788572698.587841:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572698.587848:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572698.587890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569621248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572698.587920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572698.587925:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572698.587927:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572698.587931:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9451 00000100:00080000:1.0:1788572698.587935:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572698.587940:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572698.587954:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572698.587969:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569621376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572698.587983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572698.587989:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788572698.587990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569621248:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572698.587991:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572698.588001:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9451 00000100:00080000:1.0:1788572698.588006:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572698.588011:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572698.588017:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572698.588033:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569621504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572698.588044:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572698.588048:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572698.588051:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572698.588055:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9451 00000100:00080000:1.0:1788572698.588059:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572698.588064:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572698.588071:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572698.588082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572698.588086:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569621632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572698.588098:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572698.588102:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572698.588105:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572698.588110:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9451 00000100:00080000:1.0:1788572698.588114:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572698.588119:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572698.588125:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572698.588139:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569621760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572698.588151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff920a52892680 x1875444569621248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.588176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569621248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.588188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.588193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572698.588198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569621248:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572698.588231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621376:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572698.588265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572698.588280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff920a52891500 x1875444569621376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.588300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569621376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.588308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.588313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572698.588316:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621376:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572698.588322:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572698.588342:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572698.588352:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff920a52893800 x1875444569621504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.588365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569621504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.588373:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.588377:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572698.588380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572698.588386:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621632:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572698.588405:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572698.588415:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff920a52890a80 x1875444569621632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.588427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569621632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.588440:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.588444:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572698.588448:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621632:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572698.588453:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572698.588472:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572698.588482:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572698/real 1788572698] req@ffff920a52890380 x1875444569621760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572698.588495:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569621760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572698.588503:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572698.588506:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572698.588510:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569621760:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572699.035810:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572699.611704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572700.059741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572700.635686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572701.083753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572701.659692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572702.107782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572702.683711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572703.131801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572703.707696:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572703.707704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572703.707707:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572703.707713:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572703.707725:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9457 00000100:00080000:1.0:1788572703.707735:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572703.707743:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572703.707778:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569621888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572703.707805:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572703.707810:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572703.707812:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572703.707817:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9457 00000100:00080000:1.0:1788572703.707821:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572703.707826:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572703.707836:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572703.707843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569621888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572703.707852:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569622016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572703.707865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572703.707870:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572703.707872:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572703.707877:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9457 00000100:00080000:1.0:1788572703.707882:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572703.707886:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572703.707892:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572703.707899:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572703.707921:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569622144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572703.707927:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572703/real 1788572703] req@ffff920a52890000 x1875444569621888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572703.707943:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572703.707952:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572703.707956:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572703.707962:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9457 00000100:00100000:3.0:1788572703.707965:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569621888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572703.707970:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572703.707976:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572703.707980:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788572703.707983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572703.707988:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569621888:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572703.707990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572703.708006:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569622272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572703.708008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622016:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572703.708026:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572703.708029:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572703.708032:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572703.708036:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9457 00000100:00080000:1.0:1788572703.708040:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572703.708041:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572703.708049:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572703.708055:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572703.708061:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572703/real 1788572703] req@ffff920a52893b80 x1875444569622016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572703.708073:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569622400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572703.708097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569622016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572703.708109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572703.708114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572703.708119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622016:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572703.708125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572703.708147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572703.708161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572703/real 1788572703] req@ffff920a52890380 x1875444569622144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572703.708173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569622144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572703.708186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572703.708190:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572703.708193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572703.708219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622272:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572703.708240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572703.708252:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572703/real 1788572703] req@ffff920a52890a80 x1875444569622272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572703.708262:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569622272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572703.708269:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572703.708272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572703.708275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622272:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572703.708281:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572703.708299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572703.708309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572703/real 1788572703] req@ffff920a52893800 x1875444569622400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572703.708320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569622400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572728 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572703.708327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572703.708330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572703.708334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622400:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572704.155741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572704.731739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572705.179816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572705.755713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572706.203710:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572706.779678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572707.227735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572707.803731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572708.251711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572708.827738:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572708.827740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572708.827748:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572708.827753:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572708.827762:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9462 00000100:00080000:0.0:1788572708.827771:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572708.827780:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572708.827822:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569622528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572708.827850:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572708.827853:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572708.827855:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572708.827858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9462 00000100:00080000:0.0:1788572708.827861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572708.827863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572708.827871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572708.827881:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569622656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572708.827888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572708.827891:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572708.827892:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572708.827894:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9462 00000100:00080000:0.0:1788572708.827896:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572708.827898:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572708.827900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572708.827906:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569622784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572708.827912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572708.827914:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572708.827915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572708.827918:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9462 00000100:00080000:0.0:1788572708.827920:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572708.827922:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572708.827925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572708.827931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569622912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572708.827937:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572708.827938:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572708.827944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572708.827950:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9462 00000100:00080000:0.0:1788572708.827955:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572708.827959:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572708.827965:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572708.827981:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569623040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572708.827985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569622528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572708.828123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622656:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572708.828164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572708.828199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622912:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572708.828245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623040:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572709.275736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572709.851723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572709.851892:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.202.157:988 00000400:00000100:1.0:1788572709.851931:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572709.851943:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572709.851979:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572709.851991:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572709.852002:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572709.852016:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788572709.852030:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572708/real 1788572709] req@ffff920a48353480 x1875444569622528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.852064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569622528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.852076:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.852081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572709.852086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569622528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572709.852095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572708/real 1788572709] req@ffff920a48350380 x1875444569622656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.852117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569622656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.852126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.852130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572709.852150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622656:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572709.852158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572708/real 1788572709] req@ffff920a48350700 x1875444569622784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.852171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569622784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.852179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.852183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572709.852187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572709.852194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572708/real 1788572709] req@ffff920a48351500 x1875444569622912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.852207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569622912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.852216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.852220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572709.852224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569622912:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572709.852252:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572708/real 1788572709] req@ffff920a48353800 x1875444569623040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572709.852265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569623040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572733 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572709.852273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572709.852276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572709.852280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623040:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572710.299729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572710.875711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572711.323798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572711.899721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572712.347708:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572712.923668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572713.371731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572713.947696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572713.947699:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572713.947705:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572713.947709:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572713.947719:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9467 00000100:00080000:1.0:1788572713.947728:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572713.947735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572713.947769:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569623168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572713.947800:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572713.947805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572713.947807:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572713.947813:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9467 00000100:00080000:1.0:1788572713.947818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572713.947823:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572713.947834:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572713.947840:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569623168:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572713.947849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569623296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572713.947861:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572713.947865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572713.947867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572713.947872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9467 00000100:00080000:1.0:1788572713.947878:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572713.947882:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572713.947889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572713.947903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572713.947905:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569623424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572713.947918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572713.947921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572713.947924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572713.947928:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9467 00000100:00080000:1.0:1788572713.947933:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572713.947938:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572713.947943:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572713/real 1788572713] req@ffff920a52891500 x1875444569623168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572713.947946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572713.947964:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569623552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572713.947973:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569623168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572713.947979:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572713.947988:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572713.947990:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572713.947992:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572713.947997:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9467 00000100:00080000:3.0:1788572713.948002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788572713.948004:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572713.948010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569623168:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572713.948014:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572713.948023:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572713.948036:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569623680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572713.948043:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623296:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572713.948077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572713.948092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572713/real 1788572713] req@ffff920a52892680 x1875444569623296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572713.948105:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569623296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572713.948114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572713.948118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572713.948122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623296:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572713.948128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572713.948150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572713.948162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572713/real 1788572713] req@ffff920a52891c00 x1875444569623424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572713.948175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569623424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572713.948183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572713.948187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572713.948191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572713.948197:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623552:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572713.948220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572713.948233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572713/real 1788572713] req@ffff920a52892a00 x1875444569623552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572713.948250:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569623552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572713.948260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572713.948263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572713.948267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623552:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572713.948284:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572713.948306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572713.948319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572713/real 1788572713] req@ffff920a52891f80 x1875444569623680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572713.948332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569623680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572738 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572713.948340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572713.948344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572713.948347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623680:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572714.395714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572714.971706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572715.419782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572715.995680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572716.443654:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572717.019663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572717.467683:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572718.043694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572718.491670:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572719.067660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572719.067663:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572719.067671:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572719.067674:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572719.067683:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9472 00000100:00080000:1.0:1788572719.067691:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572719.067699:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572719.067729:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569623808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572719.067752:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572719.067755:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572719.067757:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572719.067760:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9472 00000100:00080000:1.0:1788572719.067763:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572719.067766:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572719.067774:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572719.067785:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569623808:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572719.067787:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569623936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572719.067799:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572719.067802:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572719.067804:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572719.067807:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9472 00000100:00080000:1.0:1788572719.067810:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572719.067812:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572719.067816:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572719.067825:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569624064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572719.067832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572719.067836:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572719.067837:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572719.067839:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9472 00000100:00080000:1.0:1788572719.067842:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572719.067846:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788572719.067847:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788572719.067852:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572719.067862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569624192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572719.067869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572719.067872:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572719.067873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572719.067877:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9472 00000100:00100000:3.0:1788572719.067881:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff920a52893480 x1875444569623808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572719.067882:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572719.067889:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572719.067896:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572719.067906:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569624320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572719.067909:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569623808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.067931:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.067935:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572719.067938:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569623808:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572719.067956:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623936:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572719.067972:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572719.067980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff920a52890e00 x1875444569623936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.067988:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569623936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.067994:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.067997:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572719.068008:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569623936:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572719.068014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572719.068035:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572719.068047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff920a52891180 x1875444569624064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.068060:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569624064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.068074:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.068078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572719.068081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572719.068086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624192:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572719.068108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572719.068120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff920a52890700 x1875444569624192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.068130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569624192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.068138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.068141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572719.068144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624192:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572719.068150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572719.068171:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572719.068182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572719/real 1788572719] req@ffff920a52893100 x1875444569624320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572719.068194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569624320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572719.068202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572719.068205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572719.068209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624320:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572719.515752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572720.091677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572720.539708:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572721.115723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572721.563778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572722.139668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572722.587711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572723.163702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572723.611780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572724.187673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572724.187676:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572724.187693:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572724.187697:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572724.187708:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9477 00000100:00080000:1.0:1788572724.187719:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572724.187731:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572724.187768:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569624448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572724.187795:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572724.187798:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572724.187800:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572724.187803:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9477 00000100:00080000:1.0:1788572724.187806:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572724.187810:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572724.187818:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572724.187831:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569624576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572724.187835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569624448:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572724.187852:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572724.187856:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572724.187858:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572724.187863:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9477 00000100:00080000:1.0:1788572724.187867:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572724.187874:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572724.187880:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572724.187899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569624704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572724.187910:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572724.187916:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572724.187921:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572724.187923:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572724.187928:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9477 00000100:00080000:1.0:1788572724.187933:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572724.187938:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572724.187944:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572724.187956:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff920a52891880 x1875444569624448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572724.187958:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569624832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572724.187989:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572724.187992:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572724.187994:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572724.188000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569624448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572724.188001:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9477 00000100:00080000:1.0:1788572724.188008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572724.188016:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788572724.188022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788572724.188023:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788572724.188032:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572724.188039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569624448:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572724.188042:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569624960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572724.188067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624576:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.188101:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572724.188116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff920a52892d80 x1875444569624576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.188147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569624576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.188155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.188159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572724.188163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624576:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572724.188169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572724.188190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572724.188203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff920a52892300 x1875444569624704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.188213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569624704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.188221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.188224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572724.188227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572724.188232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624832:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572724.188252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572724.188262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff920a52893100 x1875444569624832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.188274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569624832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.188282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.188286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572724.188289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624832:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572724.188313:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572724.188335:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572724.188347:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572724/real 1788572724] req@ffff920a52890700 x1875444569624960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572724.188360:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569624960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572724.188369:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572724.188373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572724.188376:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569624960:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572724.635700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572725.211677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572725.659806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572726.235660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572726.683733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572727.259694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572727.707747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572728.283732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572728.731692:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572729.307735:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572729.307740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572729.307746:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572729.307753:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572729.307763:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9482 00000100:00080000:3.0:1788572729.307774:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572729.307783:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572729.307818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569625088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572729.307840:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572729.307844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572729.307848:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572729.307854:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9482 00000100:00080000:3.0:1788572729.307859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572729.307862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569625088:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572729.307868:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572729.307882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572729.307899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569625216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572729.307912:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572729.307916:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572729.307919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572729.307924:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9482 00000100:00080000:3.0:1788572729.307929:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788572729.307934:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572729.307938:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572729.307946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572729.307962:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569625344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572729.307972:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572729] req@ffff920a529b1f80 x1875444569625088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572729.307978:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572729.307987:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572729.308004:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572729.308012:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9482 00000100:00100000:1.0:1788572729.308018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569625088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572729.308019:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572729.308029:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572729.308039:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788572729.308040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572729.308049:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572729.308055:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569625088:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572729.308061:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569625472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572729.308073:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572729.308077:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572729.308095:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572729.308100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625216:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788572729.308103:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9482 00000100:00080000:3.0:1788572729.308110:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572729.308115:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572729.308121:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788572729.308138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572729.308140:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569625600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572729.308160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572729] req@ffff920a529b0000 x1875444569625216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572729.308178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569625216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572729.308188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572729.308192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572729.308196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625216:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572729.308204:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572729.308227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572729.308241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572729] req@ffff920a529b3100 x1875444569625344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572729.308262:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569625344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572729.308271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572729.308274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572729.308277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572729.308283:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625472:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572729.308305:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572729.308316:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572729] req@ffff920a529b2d80 x1875444569625472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572729.308328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569625472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572729.308337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572729.308340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572729.308343:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625472:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572729.308360:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572729.308381:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572729.308393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572729/real 1788572729] req@ffff920a529b0700 x1875444569625600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572729.308404:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569625600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572729.308419:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572729.308423:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572729.308426:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625600:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572729.755831:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572730.331747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572730.779710:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572731.355793:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572731.803771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572732.379778:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572732.827707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572733.403729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572733.851750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572734.427707:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572734.427723:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572734.427727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572734.427741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9487 00000100:00080000:1.0:1788572734.427754:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572734.427763:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572734.427811:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569625728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572734.427824:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572734.427842:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572734.427846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572734.427848:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572734.427853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9487 00000100:00080000:1.0:1788572734.427857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572734.427878:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572734.427885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569625728:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572734.427891:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572734.427909:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569625856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572734.427927:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572734.427931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572734.427934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572734.427939:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9487 00000100:00080000:1.0:1788572734.427944:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572734.427948:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572734.427954:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572734.427969:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569625984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572734.427971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572734.427983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572734.427987:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572734.427989:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572734.428002:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9487 00000100:00080000:1.0:1788572734.428007:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572734.428012:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572734.428020:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572734.428021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff920a52891180 x1875444569625728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572734.428036:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569626112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572734.428046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569625728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572734.428047:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572734.428055:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572734.428058:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572734.428065:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9487 00000100:00080000:3.0:1788572734.428069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572734.428070:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572734.428077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788572734.428078:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572734.428085:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572734.428086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569625728:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572734.428100:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569626240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572734.428123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625856:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572734.428152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572734.428168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff920a52890e00 x1875444569625856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.428183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569625856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.428191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.428194:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572734.428197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625856:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572734.428201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572734.428219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572734.428231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff920a52893480 x1875444569625984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.428243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569625984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.428252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.428255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572734.428258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569625984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572734.428264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626112:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572734.428286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572734.428298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff920a52891f80 x1875444569626112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.428318:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569626112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.428327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.428331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572734.428334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626112:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572734.428341:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572734.428363:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572734.428375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572734/real 1788572734] req@ffff920a52892a00 x1875444569626240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572734.428387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569626240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572734.428396:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572734.428400:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572734.428403:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626240:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572734.875725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572735.451770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572735.899775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572736.475759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572736.923744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572737.499870:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572737.947718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572738.523802:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572738.971742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572739.547668:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.547676:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572739.547680:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.547689:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9492 00000100:00080000:3.0:1788572739.547698:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.547705:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572739.547709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788572739.547756:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569626368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.547779:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.547782:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572739.547783:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.547787:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9492 00000100:00080000:3.0:1788572739.547790:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.547794:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572739.547803:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572739.547815:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569626496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572739.547822:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569626368:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572739.547824:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.547832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572739.547835:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.547839:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9492 00000100:00080000:3.0:1788572739.547843:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.547847:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572739.547852:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572739.547866:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569626624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.547876:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.547879:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572739.547882:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.547886:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9492 00000100:00080000:3.0:1788572739.547890:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572739.547895:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788572739.547902:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788572739.547905:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572739.547921:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569626752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572739.547931:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572739.547935:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572739.547937:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572739.547941:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9492 00000100:00080000:3.0:1788572739.547945:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572739.547951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff920a529b1c00 x1875444569626368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572739.547952:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572739.547961:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572739.547981:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569626880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572739.547986:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569626368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572739.548011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572739.548018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572739.548023:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569626368:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572739.548052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626496:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572739.548079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572739.548094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff920a529b2300 x1875444569626496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572739.548108:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569626496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572739.548122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572739.548127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572739.548131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626496:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572739.548137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572739.548160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572739.548171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff920a529b2a00 x1875444569626624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572739.548183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569626624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572739.548192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572739.548197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572739.548200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572739.548206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626752:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572739.548228:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572739.548241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff920a529b3480 x1875444569626752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572739.548253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569626752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572739.548260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572739.548263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572739.548267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626752:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572739.548274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572739.548293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572739.548303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572739/real 1788572739] req@ffff920a529b0700 x1875444569626880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572739.548315:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569626880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572739.548323:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572739.548326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572739.548330:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569626880:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572739.995829:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572740.571782:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572741.019777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572741.595736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572742.043759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572742.619749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572743.067780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572743.643774:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572744.091712:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572744.667731:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572744.667741:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572744.667746:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572744.667759:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9498 00000100:00100000:2.0:1788572744.667759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572744.667772:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572744.667780:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572744.667814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569627008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572744.667844:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572744.667849:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572744.667851:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572744.667856:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9498 00000100:00080000:1.0:1788572744.667861:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572744.667866:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572744.667877:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572744.667886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569627008:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572744.667894:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569627136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572744.667907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572744.667910:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572744.667912:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572744.667917:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9498 00000100:00080000:1.0:1788572744.667921:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572744.667926:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572744.667933:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572744.667946:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569627264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572744.667957:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572744.667961:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572744.667964:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572744.667968:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9498 00000100:00080000:1.0:1788572744.667972:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572744.667975:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572744.667981:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572744.667993:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569627392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572744.668008:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572744.668010:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627136:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572744.668015:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572744.668020:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572744.668026:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9498 00000100:00080000:1.0:1788572744.668031:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572744.668035:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572744.668041:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572744.668055:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572744.668057:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569627520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572744.668110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627392:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572744.668142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627520:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572745.115720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572745.691738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572746.139716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572746.715953:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788572746.716180:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788572746.716210:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788572746.716222:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572746.716252:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572746.716260:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572746.716266:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788572746.716278:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788572746.716303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572744/real 1788572746] req@ffff920a52890a80 x1875444569627520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.716329:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569627520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.716342:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.716348:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572746.716353:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572746.716361:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572744/real 1788572746] req@ffff920a52892680 x1875444569627136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.716374:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569627136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.716383:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.716387:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572746.716391:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627136:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572746.716398:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572744/real 1788572746] req@ffff920a52891500 x1875444569627264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.716410:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569627264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.716418:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.716422:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572746.716426:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572746.716433:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572744/real 1788572746] req@ffff920a52893800 x1875444569627392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.716446:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569627392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.716455:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.716458:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572746.716461:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627392:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572746.716468:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572744/real 1788572746] req@ffff920a52891c00 x1875444569627008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572746.716481:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569627008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572746.716489:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572746.716493:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572746.716496:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569627008:192.168.202.157@tcp:250:kworker.0 00010000:00010000:3.0:1788572747.163720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572747.739754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572748.187738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572748.763766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572749.211759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572749.787667:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572749.787678:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572749.787682:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572749.787693:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9503 00000100:00080000:1.0:1788572749.787703:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572749.787712:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572749.787743:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569627648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572749.787744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572749.787770:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572749.787774:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572749.787777:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572749.787782:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9503 00000100:00080000:1.0:1788572749.787786:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572749.787791:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572749.787801:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572749.787806:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569627648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572749.787813:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569627776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572749.787823:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572749.787827:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572749.787829:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572749.787832:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9503 00000100:00080000:1.0:1788572749.787836:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572749.787839:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572749.787843:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572749.787852:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569627904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572749.787860:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572749.787863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572749.787865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572749.787866:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572749.787869:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9503 00000100:00080000:1.0:1788572749.787872:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572749.787875:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572749.787880:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572749.787889:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569628032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572749.787896:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572749/real 1788572749] req@ffff920a52890380 x1875444569627648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572749.787898:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572749.787902:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572749.787904:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572749.787908:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9503 00000100:00080000:1.0:1788572749.787912:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572749.787915:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572749.787919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572749.787921:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569627648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572749.787929:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569628160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572749.787933:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572749.787937:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572749.787941:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569627648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572749.787962:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627776:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572749.787986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572749.787996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572749/real 1788572749] req@ffff920a52893b80 x1875444569627776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572749.788005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569627776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572749.788011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572749.788014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572749.788016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627776:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572749.788019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572749.788032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572749.788038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572749/real 1788572749] req@ffff920a52890000 x1875444569627904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572749.788046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569627904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572749.788055:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572749.788057:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572749.788059:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569627904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572749.788062:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628032:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572749.788072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572749.788078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572749/real 1788572749] req@ffff920a52891c00 x1875444569628032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572749.788084:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569628032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572749.788089:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572749.788091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572749.788093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628032:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572749.788096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572749.788106:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572749.788111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572749/real 1788572749] req@ffff920a52893800 x1875444569628160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572749.788117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569628160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572774 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572749.788122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572749.788124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572749.788126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628160:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572750.235722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572750.811706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572751.259734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572751.835693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572752.283747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572752.859687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572753.307715:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572753.883680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572754.331758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572754.907727:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572754.907731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572754.907747:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572754.907751:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572754.907762:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9508 00000100:00080000:1.0:1788572754.907771:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572754.907779:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572754.907813:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569628288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572754.907840:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572754.907844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572754.907846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572754.907851:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9508 00000100:00080000:1.0:1788572754.907855:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572754.907861:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572754.907872:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572754.907887:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569628416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572754.907896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569628288:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572754.907912:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572754.907917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572754.907919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572754.907925:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9508 00000100:00080000:1.0:1788572754.907929:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572754.907933:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572754.907940:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572754.907958:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569628544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572754.907966:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572754.907971:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572754.907973:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572754.907977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9508 00000100:00080000:1.0:1788572754.907981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572754.907989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572754.907995:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572754.908002:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572754.908015:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569628672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572754.908024:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572754.908027:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572754.908030:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572754.908036:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572754/real 1788572754] req@ffff920a52891500 x1875444569628288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572754.908038:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9508 00000100:00080000:1.0:1788572754.908046:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572754.908058:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572754.908070:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572754.908088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569628288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572754.908089:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569628800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572754.908159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572754.908169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572754.908174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569628288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572754.908205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628416:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572754.908226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572754.908237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572754/real 1788572754] req@ffff920a52892680 x1875444569628416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572754.908247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569628416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572754.908256:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572754.908259:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572754.908263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628416:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572754.908271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572754.908294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572754.908308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572754/real 1788572754] req@ffff920a52890a80 x1875444569628544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572754.908322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569628544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572754.908331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572754.908335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572754.908339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572754.908346:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628672:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572754.908367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572754.908378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572754/real 1788572754] req@ffff920a52892a00 x1875444569628672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572754.908390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569628672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572754.908399:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572754.908402:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572754.908406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628672:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572754.908411:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572754.908427:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572754.908435:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572754/real 1788572754] req@ffff920a52891f80 x1875444569628800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572754.908447:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569628800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572779 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572754.908452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572754.908454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572754.908456:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569628800:192.168.202.157@tcp:8:kworker.0 00010000:00010000:3.0:1788572755.355700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572755.931670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572756.379733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572756.955697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572757.403739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572757.979770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572758.427825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572759.003689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572759.451706:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572760.027720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572760.027737:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572760.027750:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572760.027756:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572760.027772:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9513 00000100:00080000:0.0:1788572760.027786:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572760.027797:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572760.027853:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569628928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572760.027890:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572760.027893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572760.027895:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572760.027899:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9513 00000100:00080000:0.0:1788572760.027903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572760.027909:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572760.027920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572760.027937:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569628928:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572760.027946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569629056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572760.027960:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572760.027965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572760.027967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572760.027972:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9513 00000100:00080000:0.0:1788572760.027976:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572760.027981:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572760.027987:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572760.028000:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569629184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572760.028011:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572760.028014:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572760.028016:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572760.028020:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9513 00000800:00000100:3.0:1788572760.028027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572760.028028:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572760.028037:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572760.028043:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572760.028057:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569629312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572760.028068:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572760.028072:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572760.028074:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572760.028080:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff920a48351c00 x1875444569628928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572760.028081:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9513 00000100:00080000:0.0:1788572760.028088:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572760.028094:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572760.028100:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572760.028115:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569629440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572760.028121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569628928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.028144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.028150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572760.028155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569628928:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572760.028187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629056:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.028215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572760.028230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff920a48350a80 x1875444569629056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.028251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569629056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.028260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.028264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572760.028268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629056:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572760.028275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572760.028299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572760.028311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff920a48353b80 x1875444569629184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.028337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569629184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.028346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.028350:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572760.028354:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572760.028361:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629312:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572760.028384:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572760.028396:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff920a48353800 x1875444569629312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.028408:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569629312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.028422:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.028425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572760.028429:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629312:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572760.028434:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572760.028453:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572760.028463:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572760/real 1788572760] req@ffff920a48351500 x1875444569629440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572760.028474:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569629440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572760.028482:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572760.028485:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572760.028488:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629440:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572760.475760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572761.051685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788572761.499694:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572762.075677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572762.523839:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572763.099730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572763.547791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572764.123712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572764.571826:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572765.147695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572765.147702:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.147710:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572765.147715:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.147726:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9518 00000100:00080000:2.0:1788572765.147735:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.147744:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572765.147782:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569629568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.147810:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.147814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572765.147816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.147820:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9518 00000100:00080000:2.0:1788572765.147825:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.147829:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572765.147837:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572765.147849:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569629568:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572765.147854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569629696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572765.147872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.147875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572765.147877:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.147881:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9518 00000100:00080000:2.0:1788572765.147885:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572765.147888:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572765.147892:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572765.147906:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569629824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572765.147916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788572765.147931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.147936:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572765.147939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.147946:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9518 00000100:00080000:2.0:1788572765.147952:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572765.147953:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff920a60ac4a80 x1875444569629568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572765.147962:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572765.147974:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572765.147977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569629568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572765.147988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572765.147993:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572765.147996:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569629952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572765.147999:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569629568:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572765.148015:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572765.148018:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572765.148020:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572765.148025:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9518 00000100:00080000:2.0:1788572765.148029:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572765.148031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629696:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572765.148037:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572765.148045:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572765.148061:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569630080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572765.148070:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572765.148094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff920a60ac4000 x1875444569629696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572765.148118:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569629696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572765.148124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572765.148127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572765.148129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629696:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572765.148134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572765.148151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572765.148160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff920a60ac4380 x1875444569629824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572765.148175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569629824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572765.148182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572765.148184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572765.148187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572765.148191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629952:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572765.148206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572765.148217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff920a60ac4700 x1875444569629952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572765.148232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569629952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572765.148239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572765.148242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572765.148244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569629952:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572765.148258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572765.148283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572765.148295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572765/real 1788572765] req@ffff920a60ac7800 x1875444569630080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572765.148306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569630080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572765.148311:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572765.148315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572765.148318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630080:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572765.595829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572766.171742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572766.619831:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572767.195714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572767.643763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572768.219654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572768.667732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572769.243700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572769.691776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572770.267703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572770.267704:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572770.267713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572770.267717:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572770.267725:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9523 00000100:00080000:1.0:1788572770.267734:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572770.267740:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572770.267773:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569630208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572770.267794:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572770.267796:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572770.267798:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572770.267801:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9523 00000100:00080000:1.0:1788572770.267804:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572770.267807:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572770.267816:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572770.267824:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569630208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572770.267827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569630336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572770.267837:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572770.267839:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572770.267841:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572770.267844:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9523 00000100:00080000:1.0:1788572770.267846:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572770.267848:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572770.267851:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572770.267859:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569630464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572770.267866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572770.267869:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572770.267870:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572770.267873:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9523 00000100:00080000:1.0:1788572770.267875:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572770.267877:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572770.267881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572770.267884:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572770.267891:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569630592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572770.267899:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572770.267902:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572770.267904:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572770.267907:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9523 00000100:00080000:1.0:1788572770.267911:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572770.267917:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572770.267919:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff920a52893480 x1875444569630208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572770.267924:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572770.267939:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569630208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572770.267945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569630720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572770.267951:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572770.267960:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572770.267966:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569630208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572770.267986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630336:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572770.268001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572770.268008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff920a52890e00 x1875444569630336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572770.268017:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569630336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572770.268022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572770.268024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572770.268026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630336:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572770.268030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572770.268040:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572770.268046:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff920a52891180 x1875444569630464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572770.268053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569630464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572770.268063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572770.268065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572770.268067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572770.268071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630592:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572770.268082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572770.268091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff920a52890700 x1875444569630592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572770.268104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569630592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572770.268112:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572770.268115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572770.268118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630592:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572770.268125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572770.268147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572770.268155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572770/real 1788572770] req@ffff920a52893100 x1875444569630720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572770.268163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569630720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572770.268168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572770.268170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572770.268173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630720:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572770.715787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572771.291722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572771.739759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572772.315707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572772.763808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572773.339666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572773.787755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572774.363702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572774.811768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572775.387676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572775.387690:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.387695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572775.387699:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.387707:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9528 00000100:00080000:2.0:1788572775.387718:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.387725:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572775.387759:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569630848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.387779:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.387782:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572775.387784:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.387787:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9528 00000100:00080000:2.0:1788572775.387789:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.387792:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572775.387801:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572775.387812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569630848:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572775.387812:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569630976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.387827:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.387829:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572775.387830:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.387833:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9528 00000100:00080000:2.0:1788572775.387836:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.387838:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572775.387842:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572775.387851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569631104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.387858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.387861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572775.387862:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.387865:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9528 00000100:00080000:2.0:1788572775.387869:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.387871:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572775.387874:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572775.387878:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572775.387883:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569631232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572775.387891:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572775.387894:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572775.387896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572775.387899:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9528 00000100:00080000:2.0:1788572775.387901:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572775.387904:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572775.387907:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572775.387911:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff920a60ac5500 x1875444569630848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572775.387916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569631360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572775.387943:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569630848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.387952:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.387955:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572775.387958:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569630848:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572775.387976:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630976:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572775.387992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572775.388000:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff920a60ac6300 x1875444569630976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.388008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569630976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.388014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.388015:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572775.388017:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569630976:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572775.388021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572775.388032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572775.388039:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff920a60ac7100 x1875444569631104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.388046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569631104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.388051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.388053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572775.388055:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572775.388058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631232:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572775.388069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572775.388075:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff920a60ac6a00 x1875444569631232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.388082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569631232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.388088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.388091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572775.388094:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631232:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572775.388100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572775.388129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572775.388143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572775/real 1788572775] req@ffff920a60ac5180 x1875444569631360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572775.388154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569631360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572775.388160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572775.388162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572775.388164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631360:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572775.835822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572776.411735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572776.859793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572777.435674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572777.883781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572778.459665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572778.907739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572779.483693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572779.931766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572780.507692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572780.507707:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572780.507717:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572780.507721:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572780.507731:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9533 00000100:00080000:2.0:1788572780.507741:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572780.507747:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572780.507783:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569631488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572780.507808:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572780.507812:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572780.507813:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572780.507817:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9533 00000100:00080000:2.0:1788572780.507820:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572780.507823:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572780.507832:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572780.507844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569631616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572780.507853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569631488:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572780.507858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572780.507863:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572780.507865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572780.507870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9533 00000100:00080000:2.0:1788572780.507873:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572780.507876:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572780.507880:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572780.507890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569631744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572780.507897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572780.507901:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572780.507902:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572780.507906:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9533 00000100:00080000:2.0:1788572780.507913:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572780.507916:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572780.507920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572780.507938:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569631872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572780.507944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788572780.507955:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572780.507959:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572780.507961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572780.507967:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9533 00000100:00080000:2.0:1788572780.507972:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572780.507975:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572780.507979:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572780.507992:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569632000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572780.508001:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff920a60ac5180 x1875444569631488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572780.508019:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569631488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572780.508029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572780.508033:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572780.508036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569631488:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572780.508060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631616:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572780.508078:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572780.508087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff920a60ac6a00 x1875444569631616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572780.508095:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569631616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572780.508100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572780.508102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572780.508104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631616:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572780.508109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572780.508119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572780.508125:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff920a60ac7100 x1875444569631744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572780.508132:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569631744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572780.508137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572780.508139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572780.508141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572780.508144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631872:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572780.508155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572780.508161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff920a60ac6300 x1875444569631872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572780.508167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569631872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572780.508180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572780.508183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572780.508185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569631872:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572780.508189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572780.508202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572780.508208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572780/real 1788572780] req@ffff920a60ac5500 x1875444569632000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572780.508217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569632000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572780.508222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572780.508224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572780.508225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632000:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572780.955724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572781.531694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572781.979746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572782.555685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572783.003774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572783.579701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572784.027780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572784.603790:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572785.051738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572785.627658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572785.627693:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572785.627700:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572785.627705:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572785.627714:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9539 00000100:00080000:0.0:1788572785.627723:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572785.627731:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572785.627785:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569632128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572785.627815:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572785.627820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572785.627822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572785.627826:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9539 00000100:00080000:0.0:1788572785.627831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572785.627835:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572785.627847:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572785.627859:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569632128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572785.627864:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569632256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572785.627884:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572785.627889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572785.627891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572785.627897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9539 00000100:00080000:0.0:1788572785.627901:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572785.627905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572785.627911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572785.627917:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572785.627927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569632384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572785.627938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572785.627943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572785.627945:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572785.627950:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572785/real 1788572785] req@ffff920a48350700 x1875444569632128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572785.627954:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9539 00000100:00080000:0.0:1788572785.627961:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572785.627968:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572785.627974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569632128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788572785.627976:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788572785.627988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572785.627994:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572785.627996:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569632512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572785.628000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569632128:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572785.628014:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572785.628018:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572785.628021:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788572785.628024:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632256:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572785.628028:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9539 00000100:00080000:0.0:1788572785.628034:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572785.628038:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572785.628044:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572785.628057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572785.628058:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569632640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572785.628071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572785/real 1788572785] req@ffff920a48350380 x1875444569632256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572785.628083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569632256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572785.628090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572785.628093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572785.628095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632256:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572785.628100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572785.628113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572785.628119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572785/real 1788572785] req@ffff920a48353480 x1875444569632384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572785.628126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569632384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572785.628131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572785.628139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572785.628141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572785.628145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632512:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572785.628161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572785.628173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572785/real 1788572785] req@ffff920a48351500 x1875444569632512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572785.628185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569632512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572785.628194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572785.628197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572785.628201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632512:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572785.628215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572785.628230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572785.628237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572785/real 1788572785] req@ffff920a48353800 x1875444569632640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572785.628244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569632640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572785.628250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572785.628252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572785.628254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632640:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572786.075763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572786.651716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572787.099739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572787.675699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572788.123780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572788.699775:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572789.147721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572789.723740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572790.171757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572790.747701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572790.747746:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572790.747752:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572790.747756:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572790.747764:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9544 00000100:00080000:3.0:1788572790.747774:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572790.747780:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572790.747814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569632768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572790.747842:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572790.747845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572790.747846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572790.747849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9544 00000100:00080000:3.0:1788572790.747851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572790.747853:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572790.747862:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572790.747870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569632896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572790.747878:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572790.747881:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572790.747882:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572790.747886:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9544 00000100:00080000:3.0:1788572790.747887:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572790.747890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572790.747892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572790.747900:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569633024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572790.747907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572790.747911:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572790.747913:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572790.747918:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569632768:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572790.747918:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9544 00000100:00080000:3.0:1788572790.747924:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572790.747933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572790.747940:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572790.747945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569633152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572790.747951:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572790.747954:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572790.747955:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572790.747957:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9544 00000100:00080000:3.0:1788572790.747962:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572790.747964:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572790.747966:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572790.747974:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569633280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572790.748028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572790.748081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572790/real 1788572790] req@ffff920a529b2d80 x1875444569632768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572790.748105:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569632768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572790.748119:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572790.748125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572790.748130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569632768:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572790.748140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632896:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572790.748168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572790.748183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572790/real 1788572790] req@ffff920a529b3100 x1875444569632896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572790.748198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569632896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572790.748215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572790.748220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572790.748224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569632896:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572790.748255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572790.748279:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572790.748292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572790/real 1788572790] req@ffff920a529b0000 x1875444569633024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572790.748305:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569633024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572790.748315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572790.748319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572790.748323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572790.748330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633152:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572790.748352:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572790.748364:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572790/real 1788572790] req@ffff920a529b1f80 x1875444569633152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572790.748378:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569633152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572790.748387:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572790.748391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572790.748395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633152:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572790.748403:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572790.748425:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572790.748436:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572790/real 1788572790] req@ffff920a529b0e00 x1875444569633280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572790.748449:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569633280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572815 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572790.748458:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572790.748461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572790.748466:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633280:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572791.195801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572791.771685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572792.219765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572792.795676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572793.243774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572793.819700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572794.267718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572794.843662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572795.291827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572795.867695:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572795.867696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572795.867706:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572795.867710:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572795.867720:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9549 00000100:00080000:3.0:1788572795.867730:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572795.867737:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572795.867773:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569633408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572795.867800:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572795.867804:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572795.867806:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572795.867811:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9549 00000100:00080000:3.0:1788572795.867815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572795.867821:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572795.867831:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572795.867835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569633408:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572795.867847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569633536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572795.867859:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572795.867863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572795.867866:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572795.867870:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9549 00000100:00080000:3.0:1788572795.867875:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572795.867878:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572795.867883:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572795.867896:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569633664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572795.867902:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572795.867913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572795.867917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572795.867919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572795.867924:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9549 00000100:00080000:3.0:1788572795.867929:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572795.867933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572795.867939:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572795.867947:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572795/real 1788572795] req@ffff920a529b1500 x1875444569633408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572795.867952:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569633792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572795.867978:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572795.867981:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788572795.867984:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569633408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572795.867985:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572795.867995:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9549 00000100:00080000:0.0:1788572795.868001:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572795.868002:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572795.868010:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788572795.868011:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572795.868018:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569633408:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788572795.868019:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572795.868036:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569633920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572795.868044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633536:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572795.868079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572795.868090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572795/real 1788572795] req@ffff920a529b1180 x1875444569633536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572795.868098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569633536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572795.868104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572795.868106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572795.868109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633536:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572795.868112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572795.868124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572795.868130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572795/real 1788572795] req@ffff920a529b0380 x1875444569633664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572795.868148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569633664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572795.868157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572795.868160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572795.868163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572795.868170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633792:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572795.868191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572795.868203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572795/real 1788572795] req@ffff920a529b3b80 x1875444569633792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572795.868213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569633792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572795.868219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572795.868221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572795.868223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572795.868235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572795.868248:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572795.868263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572795/real 1788572795] req@ffff920a529b0e00 x1875444569633920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572795.868271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569633920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572820 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572795.868276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572795.868278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572795.868280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569633920:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572796.315772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572796.891747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572797.339784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572797.915737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572798.363761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572798.939891:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572799.387783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572799.963696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572800.411701:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788572800.987687:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572800.987697:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572800.987701:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572800.987712:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9554 00000100:00080000:3.0:1788572800.987722:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572800.987726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572800.987737:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572800.987770:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569634048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572800.987802:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572800.987806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572800.987808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572800.987813:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9554 00000100:00080000:3.0:1788572800.987817:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572800.987822:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572800.987830:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572800.987836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569634048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572800.987847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569634176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572800.987858:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572800.987862:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572800.987864:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572800.987869:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9554 00000100:00080000:3.0:1788572800.987872:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572800.987876:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572800.987881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572800.987893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569634304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572800.987901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572800.987907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572800.987912:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572800.987914:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572800.987918:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9554 00000100:00080000:3.0:1788572800.987922:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572800.987926:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572800.987932:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572800.987942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569634432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572800.987945:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572800/real 1788572800] req@ffff920a529b1f80 x1875444569634048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572800.987967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569634048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572800.987979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572800.987983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572800.987987:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569634048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572800.988008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634176:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572800.988026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572800.988039:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572800.988041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572800/real 1788572800] req@ffff920a529b0000 x1875444569634176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572800.988045:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572800.988049:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572800.988057:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9554 00000100:00080000:3.0:1788572800.988062:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788572800.988066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569634176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572800.988078:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572800.988081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572800.988085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572800.988090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572800.988109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572800.988117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572800/real 1788572800] req@ffff920a529b3100 x1875444569634304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572800.988124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569634304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572800.988134:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572800.988137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572800.988139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572800.988150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634432:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572800.988167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572800.988178:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572800.988181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572800/real 1788572800] req@ffff920a529b2d80 x1875444569634432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788572800.988186:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572800.988201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569634432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572800.988205:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569634560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572800.988218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572800.988223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572800.988227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572800.988279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572800.988306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572800.988319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572800/real 1788572800] req@ffff920a529b0700 x1875444569634560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572800.988330:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569634560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572825 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572800.988336:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572800.988338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572800.988342:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634560:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572801.435766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572802.011679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572802.459697:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572803.035729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572803.483724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572804.059712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572804.507794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572805.083683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572805.531776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572806.107680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572806.107683:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.107691:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572806.107694:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.107703:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9559 00000100:00080000:0.0:1788572806.107711:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.107718:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572806.107751:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569634688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.107776:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.107779:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572806.107780:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.107783:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9559 00000100:00080000:0.0:1788572806.107786:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.107789:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572806.107797:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572806.107808:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569634816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572806.107808:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569634688:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572806.107822:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.107825:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572806.107826:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.107829:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9559 00000100:00080000:0.0:1788572806.107831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.107833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572806.107837:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572806.107847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569634944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.107854:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.107856:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572806.107857:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.107860:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9559 00000100:00080000:0.0:1788572806.107864:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.107866:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788572806.107871:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788572806.107873:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572806.107885:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569635072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572806.107896:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572806.107900:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572806.107903:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572806.107908:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9559 00000100:00080000:0.0:1788572806.107912:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572806.107917:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572806.107923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572806.107929:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff920a48353b80 x1875444569634688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572806.107940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569635200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572806.107952:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569634688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.107961:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.107965:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572806.107968:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569634688:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572806.107987:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634816:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572806.108004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572806.108013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff920a48350a80 x1875444569634816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.108021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569634816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.108027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.108029:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572806.108031:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572806.108035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572806.108045:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572806.108051:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff920a48351c00 x1875444569634944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.108058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569634944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.108064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.108065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572806.108067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569634944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572806.108070:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635072:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572806.108081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572806.108086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff920a48353800 x1875444569635072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.108093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569635072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.108106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.108108:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572806.108110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572806.108114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572806.108125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572806.108131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572806/real 1788572806] req@ffff920a48351500 x1875444569635200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572806.108139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569635200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572806.108145:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572806.108146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572806.108148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635200:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572806.555793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572807.131748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572807.579839:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572808.155709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572808.603793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572809.179745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572809.627758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572810.203749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572810.651815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572811.227713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572811.227772:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.227785:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572811.227791:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.227805:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9564 00000100:00080000:3.0:1788572811.227818:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.227828:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572811.227882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569635328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.227918:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.227923:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572811.227925:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.227930:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9564 00000100:00080000:3.0:1788572811.227933:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.227938:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572811.227952:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572811.227968:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569635456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.227979:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572811.227983:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569635328:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572811.227983:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572811.227985:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.227993:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9564 00000100:00080000:3.0:1788572811.227996:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.227999:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572811.228004:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572811.228015:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569635584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.228021:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.228025:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572811.228026:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.228029:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9564 00000100:00080000:3.0:1788572811.228031:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.228033:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572811.228037:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572811.228043:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569635712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572811.228049:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572811.228052:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572811.228054:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572811.228056:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9564 00000100:00080000:3.0:1788572811.228059:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572811.228062:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572811.228065:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572811.228074:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569635840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572811.228113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572811.228153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572811.228183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572811.228204:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635840:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572811.675791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572812.251774:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572812.699741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572813.275805:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572813.723804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572814.299873:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788572814.299998:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788572814.300050:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788572814.300066:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572814.300099:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572814.300107:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572814.300118:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788572814.300127:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788572814.300151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572814] req@ffff920a529b2a00 x1875444569635456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572814.300180:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569635456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572814.300192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572814.300198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572814.300202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572814.300209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572814] req@ffff920a529b2300 x1875444569635584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572814.300221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569635584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572814.300237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572814.300241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572814.300244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572814.300251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572814] req@ffff920a529b1c00 x1875444569635712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572814.300261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569635712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572814.300268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572814.300272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572814.300280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572814.300287:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572814] req@ffff920a529b0700 x1875444569635840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572814.300300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569635840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572814.300307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572814.300311:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572814.300314:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569635840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572814.300321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572811/real 1788572814] req@ffff920a529b3480 x1875444569635328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572814.300333:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569635328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572814.300341:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572814.300344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572814.300347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569635328:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788572814.747812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572815.324126:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572815.771753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572816.347738:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.347751:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572816.347756:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572816.347763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572816.347770:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9569 00000100:00080000:1.0:1788572816.347785:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.347792:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572816.347830:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569635968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.347871:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.347876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572816.347878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.347883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9569 00000100:00080000:1.0:1788572816.347888:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.347894:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572816.347903:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572816.347918:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569636096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572816.347928:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569635968:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572816.347935:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572816.347947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572816.347950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.347956:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9569 00000100:00080000:1.0:1788572816.347961:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.347966:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572816.347971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572816.347983:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569636224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.347992:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572816.347998:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572816.347999:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572816.348004:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.348011:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9569 00000100:00080000:1.0:1788572816.348018:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572816.348022:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572816.348029:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572816.348037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff920a52892300 x1875444569635968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572816.348044:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569636352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572816.348063:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572816.348069:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569635968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572816.348070:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572816.348074:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572816.348084:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9569 00000100:00080000:3.0:1788572816.348091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572816.348092:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572816.348099:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788572816.348103:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572816.348108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569635968:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572816.348113:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572816.348128:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569636480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572816.348146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636096:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572816.348174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572816.348188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff920a52892d80 x1875444569636096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.348202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569636096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.348225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.348229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572816.348233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636096:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572816.348239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572816.348262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572816.348273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff920a52891880 x1875444569636224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.348285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569636224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.348294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.348302:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572816.348306:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572816.348312:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636352:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572816.348332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572816.348343:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff920a52893100 x1875444569636352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.348362:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569636352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.348370:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.348378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572816.348382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572816.348388:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572816.348408:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572816.348418:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572816/real 1788572816] req@ffff920a52890700 x1875444569636480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572816.348430:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569636480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572816.348437:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572816.348440:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572816.348443:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572816.795807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572817.371775:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572817.819738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572818.395685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572818.843773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572819.419679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572819.867736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572820.443697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572820.891766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572821.467663:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.467671:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572821.467675:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572821.467678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572821.467684:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9574 00000100:00080000:0.0:1788572821.467693:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.467701:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572821.467732:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569636608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.467754:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.467757:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572821.467758:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.467761:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9574 00000100:00080000:0.0:1788572821.467764:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.467766:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572821.467774:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572821.467782:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569636736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.467802:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.467805:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572821.467806:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.467810:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9574 00000100:00080000:0.0:1788572821.467815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572821.467815:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569636608:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572821.467821:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572821.467828:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572821.467837:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569636864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.467843:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.467845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572821.467847:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.467850:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9574 00000100:00080000:0.0:1788572821.467853:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572821.467855:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572821.467859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572821.467868:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569636992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572821.467874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572821.467877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572821.467878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572821.467880:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9574 00000100:00080000:0.0:1788572821.467882:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788572821.467887:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572821.467888:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572821.467893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572821.467902:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569637120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572821.467922:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff920a48353480 x1875444569636608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572821.467935:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569636608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572821.467947:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572821.467950:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572821.467953:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569636608:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572821.467958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636736:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572821.467973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572821.467982:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff920a48350380 x1875444569636736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572821.467989:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569636736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572821.467994:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572821.467996:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572821.467998:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636736:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572821.468015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572821.468027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572821.468032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff920a48350700 x1875444569636864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572821.468040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569636864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572821.468045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572821.468047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572821.468049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572821.468053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636992:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572821.468063:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572821.468069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff920a48351500 x1875444569636992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572821.468076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569636992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572821.468081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572821.468083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572821.468085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569636992:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572821.468088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572821.468098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572821.468103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572821/real 1788572821] req@ffff920a48353800 x1875444569637120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572821.468109:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569637120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572821.468114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572821.468116:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572821.468118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637120:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572821.915749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572822.491735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572822.939770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572823.515660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572823.963733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572824.539712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572824.987811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572825.563734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572826.011761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572826.587740:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572826.587745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572826.587759:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572826.587765:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572826.587778:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9579 00000100:00080000:1.0:1788572826.587792:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572826.587801:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572826.587849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569637248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572826.587923:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572826.587929:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572826.587931:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572826.587938:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9579 00000100:00080000:1.0:1788572826.587943:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572826.587950:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572826.587952:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569637248:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572826.587963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572826.587984:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569637376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572826.587997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572826.588002:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572826.588004:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572826.588009:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9579 00000100:00080000:1.0:1788572826.588014:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572826.588018:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572826.588024:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788572826.588036:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572826.588039:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569637504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572826.588051:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572826.588055:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572826.588057:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572826.588062:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9579 00000100:00080000:1.0:1788572826.588068:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572826.588072:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572826.588078:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572826.588094:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569637632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572826.588095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff920a52891180 x1875444569637248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572826.588114:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572826.588121:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572826.588123:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572826.588128:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9579 00000100:00080000:1.0:1788572826.588131:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572826.588136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569637248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572826.588139:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572826.588148:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788572826.588157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788572826.588163:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569637760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572826.588167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572826.588176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569637248:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572826.588212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637376:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572826.588237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572826.588248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff920a52890e00 x1875444569637376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.588257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569637376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.588263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.588265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572826.588268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637376:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572826.588278:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572826.588291:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572826.588297:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff920a52893480 x1875444569637504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.588305:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569637504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.588314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.588316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572826.588319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572826.588322:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637632:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572826.588334:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572826.588339:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff920a52891f80 x1875444569637632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.588346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569637632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.588351:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.588353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572826.588355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637632:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572826.588358:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572826.588368:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572826.588374:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572826/real 1788572826] req@ffff920a52892a00 x1875444569637760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572826.588381:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569637760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572826.588385:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572826.588387:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572826.588389:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569637760:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572827.035811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572827.611755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572828.059734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572828.635718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572829.083776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572829.659750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572830.107753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572830.683686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572831.131703:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572831.707664:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572831.707672:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572831.707676:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572831.707691:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9585 00000100:00100000:3.0:1788572831.707694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572831.707702:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572831.707710:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572831.707739:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569637888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572831.707761:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572831.707764:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572831.707766:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572831.707769:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9585 00000100:00080000:1.0:1788572831.707772:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572831.707775:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572831.707784:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572831.707793:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569638016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572831.707801:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569637888:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572831.707803:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572831.707808:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572831.707811:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572831.707816:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9585 00000100:00080000:1.0:1788572831.707819:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572831.707821:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572831.707824:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572831.707833:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569638144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572831.707840:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572831.707843:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572831.707845:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572831.707847:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9585 00000100:00080000:1.0:1788572831.707851:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572831.707853:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572831.707857:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572831.707864:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569638272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572831.707870:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572831.707874:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572831.707877:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788572831.707878:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572831.707881:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9585 00000100:00080000:1.0:1788572831.707885:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572831.707887:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572831.707890:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572831.707897:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569638400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572831.707920:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572831/real 1788572831] req@ffff920a52890a80 x1875444569637888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572831.707941:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569637888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572831.707962:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572831.707966:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572831.707971:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569637888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572831.708003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638016:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572831.708027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572831.708047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572831/real 1788572831] req@ffff920a52892680 x1875444569638016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572831.708061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569638016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572831.708070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572831.708074:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572831.708078:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638016:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572831.708084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572831.708108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572831.708120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572831/real 1788572831] req@ffff920a52891500 x1875444569638144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572831.708133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569638144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572831.708142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572831.708146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572831.708149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572831.708156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638272:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572831.708176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572831.708187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572831/real 1788572831] req@ffff920a52893800 x1875444569638272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572831.708199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569638272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572831.708208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572831.708211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572831.708215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638272:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572831.708221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572831.708244:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572831.708256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572831/real 1788572831] req@ffff920a52891c00 x1875444569638400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572831.708277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569638400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572856 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572831.708286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572831.708289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572831.708293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638400:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572832.155689:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572832.731662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572833.179754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572833.755680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572834.203769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572834.779715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572835.227793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572835.803715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572836.251813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572836.827790:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572836.827824:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572836.827840:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572836.827848:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572836.827864:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9590 00000100:00080000:1.0:1788572836.827883:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572836.827894:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572836.827971:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569638528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572836.828002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572836.828006:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572836.828008:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572836.828017:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9590 00000100:00080000:1.0:1788572836.828041:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572836.828054:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572836.828077:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572836.828142:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569638656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572836.828158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569638528:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572836.828159:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572836.828171:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572836.828176:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572836.828183:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9590 00000100:00080000:1.0:1788572836.828189:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572836.828195:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572836.828216:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572836.828240:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569638784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572836.828250:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572836.828254:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572836.828256:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572836.828260:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9590 00000100:00080000:1.0:1788572836.828264:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572836.828267:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572836.828273:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572836.828282:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569638912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572836.828291:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572836.828294:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572836.828296:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572836.828299:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9590 00000100:00080000:1.0:1788572836.828302:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572836.828306:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572836.828328:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572836.828343:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569639040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788572836.828692:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572836.828816:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572836/real 1788572836] req@ffff920a52890000 x1875444569638528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572836.828851:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569638528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572836.828865:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572836.828872:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572836.828877:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569638528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572836.828909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638656:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572836.828946:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572836.828983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572836/real 1788572836] req@ffff920a52893b80 x1875444569638656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572836.829001:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569638656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572836.829014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572836.829019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572836.829024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638656:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572836.829031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572836.829052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572836.829063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572836/real 1788572836] req@ffff920a52890380 x1875444569638784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572836.829075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569638784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572836.829087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572836.829107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572836.829113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572836.829121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638912:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572836.829150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572836.829167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572836/real 1788572836] req@ffff920a52891c00 x1875444569638912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572836.829178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569638912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572836.829185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572836.829188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572836.829192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569638912:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572836.829198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572836.829215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572836.829224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572836/real 1788572836] req@ffff920a52893800 x1875444569639040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572836.829238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569639040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572861 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572836.829248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572836.829253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572836.829257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639040:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572837.275751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572837.851724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572838.299805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572838.875753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572839.323773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572839.899683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572840.347790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572840.923651:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572841.371722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572841.947697:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572841.947702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572841.947711:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572841.947716:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572841.947727:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9595 00000100:00080000:1.0:1788572841.947737:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572841.947744:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572841.947782:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569639168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572841.947809:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572841.947813:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572841.947815:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572841.947820:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9595 00000100:00080000:1.0:1788572841.947824:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572841.947828:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572841.947839:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572841.947854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569639296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572841.947865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572841.947868:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572841.947870:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572841.947873:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9595 00000100:00080000:1.0:1788572841.947877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572841.947880:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572841.947884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572841.947891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569639168:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572841.947894:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569639424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572841.947904:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572841.947907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572841.947908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572841.947911:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9595 00000100:00080000:1.0:1788572841.947914:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572841.947917:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572841.947921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572841.947931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569639552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572841.947938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572841.947940:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572841.947941:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572841.947944:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9595 00000100:00080000:1.0:1788572841.947946:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572841.947948:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572841.947951:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572841.947954:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572841.947962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569639680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572841.947989:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572841/real 1788572841] req@ffff920a52891500 x1875444569639168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572841.948002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569639168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572841.948010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572841.948013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572841.948016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569639168:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572841.948021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639296:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572841.948035:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572841.948043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572841/real 1788572841] req@ffff920a52892680 x1875444569639296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572841.948051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569639296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572841.948056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572841.948058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572841.948060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639296:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572841.948076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572841.948094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572841.948104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572841/real 1788572841] req@ffff920a52890a80 x1875444569639424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572841.948117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569639424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572841.948126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572841.948130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572841.948133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572841.948139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639552:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572841.948159:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572841.948170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572841/real 1788572841] req@ffff920a52892a00 x1875444569639552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572841.948182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569639552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572841.948192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572841.948230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572841.948233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639552:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572841.948237:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572841.948252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572841.948260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572841/real 1788572841] req@ffff920a52891f80 x1875444569639680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572841.948268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569639680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572866 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572841.948273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572841.948275:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572841.948277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639680:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572842.395764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572842.971747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572843.419799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572843.995804:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572844.443776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572845.019667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572845.467752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572846.043695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572846.491727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572847.067678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572847.067683:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.067694:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572847.067700:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.067713:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9600 00000100:00080000:3.0:1788572847.067727:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.067738:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572847.067789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569639808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.067823:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.067827:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572847.067830:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.067835:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9600 00000100:00080000:3.0:1788572847.067840:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.067845:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572847.067854:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569639808:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788572847.067859:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572847.067876:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569639936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.067888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.067892:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572847.067895:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.067900:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9600 00000100:00080000:3.0:1788572847.067923:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.067928:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572847.067934:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572847.067939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572847.067950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569640064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.067961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.067966:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572847.067968:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572847.067972:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9600 00000100:00080000:3.0:1788572847.067976:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.067980:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572847.067987:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572847.067989:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff920a529b2d80 x1875444569639808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572847.068000:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569640192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572847.068012:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572847.068016:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572847.068018:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572847.068021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569639808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572847.068024:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9600 00000100:00080000:3.0:1788572847.068030:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572847.068038:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788572847.068044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788572847.068044:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572847.068052:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572847.068059:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569640320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572847.068060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569639808:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572847.068096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639936:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572847.068125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572847.068139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff920a529b3100 x1875444569639936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572847.068153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569639936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572847.068159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572847.068162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572847.068164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569639936:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572847.068169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572847.068184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572847.068191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff920a529b0000 x1875444569640064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572847.068200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569640064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572847.068205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572847.068207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572847.068209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572847.068213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640192:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572847.068230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572847.068251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff920a529b1f80 x1875444569640192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572847.068269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569640192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572847.068277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572847.068279:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572847.068282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640192:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572847.068286:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572847.068330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572847.068350:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572847/real 1788572847] req@ffff920a529b0e00 x1875444569640320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572847.068361:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569640320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572847.068376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572847.068380:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572847.068383:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640320:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572847.515793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572848.091683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572848.539772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572849.115695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572849.563746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572850.139684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572850.587772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572851.163638:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572851.611719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572852.187697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572852.187714:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.187722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572852.187727:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.187736:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9605 00000100:00080000:0.0:1788572852.187746:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.187754:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572852.187792:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569640448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.187828:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.187831:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572852.187833:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.187836:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9605 00000100:00080000:0.0:1788572852.187840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.187843:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572852.187853:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572852.187864:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569640448:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572852.187880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569640576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.187894:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.187898:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572852.187901:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.187904:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9605 00000100:00080000:0.0:1788572852.187908:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.187912:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572852.187916:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572852.187928:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569640704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572852.187939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572852.187942:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572852.187947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572852.187950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.187956:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9605 00000100:00080000:0.0:1788572852.187962:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.187968:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572852.187974:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572852.187983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff920a48351c00 x1875444569640448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572852.187987:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569640832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572852.188024:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572852.188025:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569640448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572852.188028:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572852.188031:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572852.188038:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9605 00000100:00080000:0.0:1788572852.188044:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572852.188050:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788572852.188053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788572852.188054:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572852.188064:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572852.188069:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569640960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572852.188070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569640448:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572852.188098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640576:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572852.188122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572852.188147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff920a48350a80 x1875444569640576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572852.188164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569640576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572852.188173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572852.188177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572852.188181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640576:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572852.188187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572852.188211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572852.188223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff920a48353b80 x1875444569640704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572852.188233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569640704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572852.188239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572852.188241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572852.188243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572852.188248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640832:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572852.188262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572852.188269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff920a48353800 x1875444569640832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572852.188276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569640832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572852.188286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572852.188288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572852.188290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640832:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572852.188294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572852.188304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572852.188309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572852/real 1788572852] req@ffff920a48351500 x1875444569640960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572852.188316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569640960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572852.188321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572852.188323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572852.188325:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569640960:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572852.635725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572853.211688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572853.659707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572854.235697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572854.683707:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572855.259687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572855.707807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572856.283678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572856.731737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572857.307705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572857.307760:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572857.307768:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572857.307773:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572857.307784:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9610 00000100:00080000:0.0:1788572857.307795:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572857.307804:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572857.307843:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569641088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572857.307887:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572857.307894:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572857.307896:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572857.307900:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9610 00000100:00080000:0.0:1788572857.307905:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572857.307910:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572857.307920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572857.307929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569641088:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572857.307937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569641216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572857.307951:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572857.307955:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572857.307957:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572857.307962:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9610 00000100:00080000:0.0:1788572857.307966:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572857.307971:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572857.307977:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572857.307985:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572857.307995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569641344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572857.308007:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572857.308011:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572857.308014:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572857.308016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff920a48350700 x1875444569641088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.308022:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9610 00000100:00080000:0.0:1788572857.308031:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572857.308040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569641088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572857.308041:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572857.308052:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572857.308056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572857.308063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572857.308070:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569641472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572857.308071:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569641088:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572857.308095:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572857.308099:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572857.308101:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572857.308106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641216:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572857.308109:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9610 00000100:00080000:0.0:1788572857.308115:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572857.308120:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572857.308127:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572857.308133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572857.308147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff920a48350380 x1875444569641216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572857.308161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569641216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572857.308167:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569641600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572857.308176:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572857.308186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572857.308192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641216:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572857.308200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572857.308222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572857.308234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff920a48353480 x1875444569641344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572857.308245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569641344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572857.308253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572857.308256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572857.308260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572857.308266:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641472:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572857.308287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572857.308298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff920a48351500 x1875444569641472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572857.308310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569641472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572857.308317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572857.308321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572857.308324:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641472:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572857.308345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572857.308367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572857.308380:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572857/real 1788572857] req@ffff920a48353800 x1875444569641600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572857.308393:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569641600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572857.308402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572857.308406:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572857.308410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641600:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572857.755762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572858.331743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572858.779703:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572859.355698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572859.803794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572860.379678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572860.827813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572861.403697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572861.851828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788572862.427721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572862.427729:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572862.427738:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572862.427743:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572862.427754:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9615 00000100:00080000:0.0:1788572862.427765:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572862.427773:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572862.427817:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569641728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572862.427844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572862.427847:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572862.427849:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572862.427852:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9615 00000100:00080000:0.0:1788572862.427856:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572862.427860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572862.427870:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572862.427879:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569641856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572862.427888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572862.427892:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572862.427898:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572862.427899:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569641728:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572862.427904:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9615 00000100:00080000:0.0:1788572862.427917:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572862.427920:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572862.427923:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572862.427937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569641984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572862.427944:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572862.427946:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572862.427948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572862.427951:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9615 00000100:00080000:0.0:1788572862.427954:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572862.427958:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572862.427961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572862.427970:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569642112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572862.427982:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572862.427987:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1788572862.427987:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572862.427992:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572862.427996:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9615 00000100:00080000:0.0:1788572862.427999:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572862.428003:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572862.428007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572862.428018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569642240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572862.428032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572862] req@ffff920a48353b80 x1875444569641728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572862.428049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569641728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572862.428058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572862.428061:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572862.428064:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569641728:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572862.428085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641856:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572862.428103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572862.428111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572862] req@ffff920a48350a80 x1875444569641856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572862.428119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569641856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572862.428125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572862.428127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572862.428129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641856:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572862.428133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572862.428143:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572862.428150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572862] req@ffff920a48351c00 x1875444569641984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572862.428157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569641984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572862.428162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572862.428164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572862.428166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569641984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572862.428169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642112:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572862.428179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572862.428185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572862] req@ffff920a48353800 x1875444569642112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572862.428191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569642112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572862.428196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572862.428198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572862.428200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642112:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572862.428203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572862.428213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572862.428218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572862/real 1788572862] req@ffff920a48351500 x1875444569642240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572862.428225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569642240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572862.428230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572862.428233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572862.428235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642240:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572862.875777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572863.451732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572863.899767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572864.475717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572864.923788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572865.499710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572865.947788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572866.523702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572866.971773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572867.547698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572867.547732:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572867.547740:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572867.547745:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572867.547755:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9620 00000100:00080000:1.0:1788572867.547765:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572867.547774:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572867.547814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569642368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572867.547848:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572867.547853:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572867.547855:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572867.547860:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9620 00000100:00080000:1.0:1788572867.547864:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572867.547869:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572867.547880:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572867.547893:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569642368:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572867.547899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569642496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572867.547914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572867.547918:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572867.547922:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572867.547927:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9620 00000100:00080000:1.0:1788572867.547932:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572867.547937:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572867.547943:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788572867.547951:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572867.547960:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569642624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572867.547972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572867.547977:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572867.547979:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572867.547983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff920a52893480 x1875444569642368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572867.547987:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9620 00000100:00080000:1.0:1788572867.547996:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572867.548003:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572867.548005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569642368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572867.548012:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572867.548016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572867.548020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572867.548023:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569642368:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572867.548030:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569642752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572867.548044:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572867.548045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642496:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572867.548055:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572867.548058:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572867.548063:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9620 00000100:00080000:1.0:1788572867.548068:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572867.548073:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788572867.548077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788572867.548082:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572867.548096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff920a52890e00 x1875444569642496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572867.548102:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569642880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572867.548128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569642496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572867.548136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572867.548140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572867.548143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642496:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572867.548149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572867.548167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572867.548176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff920a52891180 x1875444569642624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572867.548184:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569642624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572867.548189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572867.548191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572867.548193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572867.548202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642752:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572867.548214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572867.548220:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff920a52890700 x1875444569642752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572867.548227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569642752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572867.548232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572867.548234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572867.548236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642752:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572867.548240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572867.548250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572867.548256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572867/real 1788572867] req@ffff920a52893100 x1875444569642880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572867.548263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569642880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572867.548267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572867.548269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572867.548271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569642880:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572867.995792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572868.571706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572869.019774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572869.595691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572870.043735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572870.619651:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572871.067720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572871.643676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572872.091759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572872.667653:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572872.668091:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572872.668096:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572872.668100:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572872.668107:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9626 00000100:00080000:0.0:1788572872.668116:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572872.668122:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572872.668149:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569643008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572872.668177:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572872.668180:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572872.668183:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572872.668186:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9626 00000100:00080000:0.0:1788572872.668189:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572872.668192:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572872.668200:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572872.668209:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569643136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572872.668218:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572872.668222:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572872.668225:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788572872.668226:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569643008:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572872.668230:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9626 00000100:00080000:0.0:1788572872.668236:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572872.668240:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572872.668243:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572872.668251:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569643264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572872.668258:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572872.668269:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572872.668271:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572872.668276:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9626 00000100:00080000:0.0:1788572872.668281:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572872.668286:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572872.668292:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572872.668299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572872.668311:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569643392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572872.668324:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572872.668328:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572872.668331:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572872.668339:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9626 00000100:00100000:2.0:1788572872.668339:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572872/real 1788572872] req@ffff920a48353480 x1875444569643008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572872.668345:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572872.668365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569643008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572872.668376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572872.668381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572872.668385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569643008:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572872.668413:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643136:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572872.668430:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572872.668437:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572872.668442:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572872.668457:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569643520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572872.668459:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572872/real 1788572872] req@ffff920a48350380 x1875444569643136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572872.668503:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569643136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572872.668510:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572872.668514:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572872.668517:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643136:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572872.668524:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572872.668542:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572872.668556:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572872/real 1788572872] req@ffff920a48350700 x1875444569643264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572872.668569:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569643264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572872.668577:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572872.668580:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572872.668582:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572872.668614:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643392:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572872.668632:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572872.668646:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572872/real 1788572872] req@ffff920a48351500 x1875444569643392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572872.668657:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569643392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572872.668664:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572872.668666:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572872.668668:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643392:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572872.668679:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572872.668690:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572872.668696:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572872/real 1788572872] req@ffff920a48353800 x1875444569643520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572872.668703:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569643520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572872.668708:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572872.668710:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572872.668711:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643520:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572873.115742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572873.691736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572874.139714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572874.715687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572875.163754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572875.739717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572876.187754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572876.763721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572877.211795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572877.787706:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572877.787717:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572877.787722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788572877.787725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572877.787741:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9631 00000100:00080000:0.0:1788572877.787754:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572877.787762:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572877.787812:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569643648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572877.787837:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572877.787840:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572877.787841:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572877.787844:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9631 00000100:00080000:0.0:1788572877.787847:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572877.787850:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572877.787861:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572877.787871:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569643776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572877.787878:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572877.787881:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572877.787883:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572877.787886:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9631 00000100:00080000:0.0:1788572877.787888:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572877.787893:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572877.787914:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572877.787924:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569643648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572877.787926:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569643904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572877.787953:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572877.787959:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572877.787963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572877.787968:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9631 00000100:00080000:0.0:1788572877.787973:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572877.787979:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572877.787995:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572877.788009:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569644032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572877.788018:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572877.788020:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572877.788023:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572877.788028:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9631 00000100:00080000:0.0:1788572877.788034:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788572877.788036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643776:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788572877.788040:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572877.788050:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572877.788064:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569644160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572877.788114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572877.788148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644032:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572877.788180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644160:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572878.235720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572878.811786:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572879.259785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572879.835742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572880.283713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572880.859876:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788572880.859904:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788572880.859968:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572880.859980:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572880.860021:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572880.860032:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572880.860042:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572880.860051:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788572880.860077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572877/real 1788572880] req@ffff920a48353b80 x1875444569643904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572880.860102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569643904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572880.860139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572880.860146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572880.860150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572880.860158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572877/real 1788572880] req@ffff920a48353800 x1875444569644032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572880.860171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569644032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572880.860179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572880.860183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572880.860186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644032:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572880.860193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572877/real 1788572880] req@ffff920a48351500 x1875444569644160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572880.860204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569644160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572880.860211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572880.860215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572880.860218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572880.860226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572877/real 1788572880] req@ffff920a48351c00 x1875444569643648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572880.860239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569643648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572880.860249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572880.860253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572880.860258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569643648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572880.860265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572877/real 1788572880] req@ffff920a48350a80 x1875444569643776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572880.860278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569643776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572902 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572880.860288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572880.860292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572880.860296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569643776:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788572881.307828:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572881.883754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572882.331779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572882.907727:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572882.907736:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572882.907740:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572882.907750:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9636 00000100:00080000:0.0:1788572882.907760:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572882.907766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572882.907798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569644288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572882.907798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572882.907827:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572882.907830:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572882.907832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572882.907838:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9636 00000100:00080000:0.0:1788572882.907841:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572882.907844:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572882.907853:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572882.907862:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569644416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572882.907875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569644288:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572882.907877:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572882.907888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572882.907890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572882.907895:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9636 00000100:00080000:0.0:1788572882.907899:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572882.907903:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572882.907910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572882.907926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569644544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572882.907934:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572882.907938:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572882.907939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572882.907942:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9636 00000100:00080000:0.0:1788572882.907946:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572882.907961:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572882.907968:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572882.907982:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572882.907983:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569644672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572882.907995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572882.907999:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572882.908001:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572882.908005:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9636 00000100:00080000:0.0:1788572882.908010:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572882.908014:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572882.908020:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572882.908032:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569644800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572882.908037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572882/real 1788572882] req@ffff920a48350700 x1875444569644288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572882.908077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569644288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572882.908090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572882.908096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572882.908100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569644288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572882.908129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644416:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572882.908156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572882.908170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572882/real 1788572882] req@ffff920a48350380 x1875444569644416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572882.908182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569644416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572882.908189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572882.908192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572882.908195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644416:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572882.908202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572882.908220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572882.908231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572882/real 1788572882] req@ffff920a48353480 x1875444569644544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572882.908241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569644544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572882.908249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572882.908252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572882.908255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572882.908261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644672:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572882.908281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572882.908291:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572882/real 1788572882] req@ffff920a48350a80 x1875444569644672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572882.908308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569644672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572882.908315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572882.908318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572882.908321:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644672:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572882.908326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572882.908345:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572882.908355:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572882/real 1788572882] req@ffff920a48351c00 x1875444569644800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572882.908365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569644800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572907 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572882.908372:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572882.908375:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572882.908378:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569644800:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572883.355798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572883.931851:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572884.379751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572884.955706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572885.403813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572885.979725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572886.427743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572887.003686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572887.451700:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572888.027700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572888.027811:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.027817:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572888.027821:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.027830:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9641 00000100:00080000:0.0:1788572888.027839:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.027846:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572888.027881:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569644928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572888.027906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.027910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572888.027912:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.027916:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9641 00000100:00080000:0.0:1788572888.027919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.027924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572888.027933:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572888.027942:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569644928:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572888.027948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569645056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572888.027960:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.027965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572888.027967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.027971:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9641 00000100:00080000:0.0:1788572888.027975:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.027979:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572888.027983:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572888.027994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569645184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572888.028003:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572888.028021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572888.028022:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572888.028027:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.028032:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9641 00000100:00080000:0.0:1788572888.028036:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572888.028041:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572888.028046:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572888.028059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff920a48351500 x1875444569644928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572888.028069:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569645312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572888.028079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569644928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572888.028081:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572888.028091:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572888.028094:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572888.028103:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9641 00000100:00080000:0.0:1788572888.028110:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572888.028110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572888.028116:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788572888.028117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572888.028125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569644928:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788572888.028131:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572888.028145:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569645440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572888.028151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645056:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572888.028181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572888.028196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff920a48353800 x1875444569645056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572888.028208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569645056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572888.028216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572888.028219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572888.028222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645056:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572888.028227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572888.028245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572888.028255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff920a48353b80 x1875444569645184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572888.028265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569645184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572888.028273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572888.028276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572888.028279:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572888.028284:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645312:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572888.028302:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572888.028311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff920a48351c00 x1875444569645312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572888.028321:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569645312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572888.028328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572888.028331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572888.028334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645312:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572888.028348:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572888.028367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572888.028377:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572888/real 1788572888] req@ffff920a48350a80 x1875444569645440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572888.028387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569645440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572888.028394:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572888.028397:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572888.028400:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645440:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572888.475707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572889.051697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572889.499803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572890.075710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572890.523760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572891.099702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572891.547756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572892.123709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572892.571738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572893.147729:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572893.147740:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788572893.147743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572893.147745:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572893.147760:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9646 00000100:00080000:0.0:1788572893.147770:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572893.147777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572893.147813:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569645568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572893.147839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572893.147843:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572893.147845:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572893.147849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9646 00000100:00080000:0.0:1788572893.147852:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572893.147855:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572893.147864:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572893.147875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569645696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572893.147885:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572893.147890:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788572893.147891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569645568:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572893.147892:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572893.147901:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9646 00000100:00080000:0.0:1788572893.147905:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572893.147908:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572893.147912:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572893.147922:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569645824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572893.147929:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572893.147933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572893.147934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572893.147937:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9646 00000100:00080000:0.0:1788572893.147940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572893.147944:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572893.147949:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788572893.147953:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572893.147961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569645952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572893.147970:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572893.147973:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572893.147975:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572893.147978:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9646 00000100:00080000:0.0:1788572893.147982:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788572893.147983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff920a48353480 x1875444569645568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572893.147988:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572893.147996:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788572893.148005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569645568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572893.148010:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569646080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572893.148016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.148022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572893.148027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569645568:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572893.148054:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645696:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572893.148104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572893.148116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff920a48350380 x1875444569645696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.148128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569645696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.148135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.148139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572893.148142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645696:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572893.148148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572893.148168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572893.148179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff920a48350700 x1875444569645824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.148188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569645824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.148197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.148199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572893.148203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572893.148208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645952:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572893.148227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572893.148236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff920a48350a80 x1875444569645952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.148247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569645952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.148253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.148255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572893.148257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569645952:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572893.148261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572893.148276:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572893.148286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572893/real 1788572893] req@ffff920a48351c00 x1875444569646080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572893.148296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569646080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572893.148302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572893.148305:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572893.148308:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646080:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572893.595731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572894.171762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572894.619776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572895.195699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572895.643724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572896.219696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572896.667802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572897.243683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572897.691726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572898.267706:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572898.267715:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572898.267719:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572898.267731:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9651 00000100:00100000:0.0:1788572898.267740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572898.267741:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572898.267748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572898.267782:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569646208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572898.267804:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572898.267807:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572898.267808:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572898.267811:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9651 00000100:00080000:1.0:1788572898.267814:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572898.267816:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572898.267823:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572898.267832:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569646336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572898.267840:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572898.267842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572898.267844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572898.267850:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9651 00000100:00100000:3.0:1788572898.267852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569646208:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572898.267853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572898.267858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572898.267863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572898.267872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569646464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572898.267879:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572898.267882:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572898.267883:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572898.267886:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9651 00000100:00080000:1.0:1788572898.267889:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572898.267891:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572898.267894:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572898.267902:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569646592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572898.267908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788572898.267912:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572898.267913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572898.267917:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572898.267920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9651 00000100:00080000:1.0:1788572898.267923:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572898.267926:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572898.267929:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572898.267937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569646720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788572898.267947:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff920a52891880 x1875444569646208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572898.267962:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569646208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572898.267969:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572898.267972:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572898.267975:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569646208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572898.267981:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646336:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572898.267997:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572898.268005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff920a52892d80 x1875444569646336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572898.268013:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569646336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572898.268018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572898.268020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572898.268022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646336:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572898.268039:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788572898.268050:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572898.268056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff920a52892300 x1875444569646464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572898.268064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569646464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572898.268069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572898.268071:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572898.268073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572898.268076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646592:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572898.268086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572898.268091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff920a52893100 x1875444569646592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572898.268105:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569646592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572898.268110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572898.268112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572898.268114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646592:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788572898.268117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788572898.268147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788572898.268155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572898/real 1788572898] req@ffff920a52890700 x1875444569646720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572898.268162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569646720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572898.268168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572898.268170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788572898.268172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646720:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572898.715802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572899.291718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572899.739841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572900.315701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572900.763788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572901.339678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572901.787751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572902.363705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572902.811771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572903.387712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572903.387712:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.387723:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572903.387728:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.387738:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9656 00000100:00080000:2.0:1788572903.387749:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.387758:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572903.387798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569646848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572903.387837:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.387842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572903.387844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.387849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9656 00000100:00080000:2.0:1788572903.387853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.387858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572903.387876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569646848:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788572903.387877:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572903.387899:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569646976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572903.387912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.387916:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572903.387918:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.387923:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9656 00000100:00080000:2.0:1788572903.387927:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.387940:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788572903.387946:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788572903.387948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572903.387965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569647104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572903.387977:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.387982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788572903.387983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff920a60ac7800 x1875444569646848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572903.387984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.387992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9656 00000100:00080000:2.0:1788572903.388000:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572903.388006:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572903.388011:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572903.388014:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569646848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572903.388026:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569647232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572903.388027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572903.388038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572903.388046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569646848:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572903.388049:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572903.388055:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572903.388058:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572903.388064:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9656 00000100:00080000:2.0:1788572903.388069:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572903.388074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646976:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572903.388075:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572903.388083:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572903.388101:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569647360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572903.388111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572903.388131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff920a60ac4700 x1875444569646976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572903.388145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569646976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572903.388155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572903.388159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572903.388164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569646976:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572903.388170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572903.388207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572903.388219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff920a60ac4380 x1875444569647104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572903.388232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569647104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572903.388241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572903.388245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572903.388249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572903.388255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647232:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572903.388273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572903.388283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff920a60ac4000 x1875444569647232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572903.388295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569647232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572903.388304:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572903.388307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572903.388310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647232:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572903.388336:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572903.388357:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572903.388369:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572903/real 1788572903] req@ffff920a60ac4a80 x1875444569647360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572903.388381:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569647360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572903.388388:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572903.388391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572903.388396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647360:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572903.835780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572904.411702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572904.859826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572905.435757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572905.883774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572906.459712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572906.907763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572907.483688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572907.931757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572908.507713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572908.507739:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.507747:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572908.507751:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.507760:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9661 00000100:00080000:2.0:1788572908.507771:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.507778:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572908.507818:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569647488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.507845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.507848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572908.507850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.507853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9661 00000100:00080000:2.0:1788572908.507856:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.507860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572908.507869:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572908.507878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569647616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.507887:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.507889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572908.507891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.507897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9661 00000100:00080000:2.0:1788572908.507901:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.507905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572908.507906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569647488:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788572908.507909:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572908.507931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569647744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.507940:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.507943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572908.507944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.507947:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9661 00000100:00080000:2.0:1788572908.507950:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.507953:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572908.507956:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572908.507965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569647872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572908.507972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572908.507974:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572908.507975:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572908.507977:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9661 00000100:00080000:2.0:1788572908.507982:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572908.507984:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572908.507987:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572908.507995:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569648000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788572908.508001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572908.508055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff920a60ac4e00 x1875444569647488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572908.508072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569647488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572908.508082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572908.508087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572908.508093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569647488:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572908.508127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647616:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572908.508164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572908.508183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff920a60ac5c00 x1875444569647616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572908.508198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569647616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572908.508210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572908.508216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572908.508221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647616:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572908.508246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572908.508284:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572908.508307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff920a60ac4a80 x1875444569647744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572908.508339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569647744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572908.508356:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572908.508363:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572908.508370:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572908.508378:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647872:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572908.508411:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572908.508442:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff920a60ac4000 x1875444569647872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572908.508462:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569647872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572908.508475:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572908.508481:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572908.508487:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569647872:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572908.508495:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572908.508521:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572908.508531:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572908/real 1788572908] req@ffff920a60ac4380 x1875444569648000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572908.508552:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569648000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572908.508557:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572908.508559:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572908.508561:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648000:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572908.955733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572909.531648:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572909.979736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572910.555696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572911.003721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572911.579701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572912.027780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572912.603686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572913.051736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788572913.627702:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572913.627703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572913.627713:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572913.627719:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572913.627728:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9667 00000100:00080000:2.0:1788572913.627737:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572913.627746:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572913.627779:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569648128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572913.627802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572913.627805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572913.627806:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572913.627810:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9667 00000100:00080000:2.0:1788572913.627812:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572913.627815:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572913.627823:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572913.627831:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569648256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572913.627835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569648128:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572913.627847:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572913.627850:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572913.627851:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572913.627855:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9667 00000100:00080000:2.0:1788572913.627857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572913.627859:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572913.627863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572913.627875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569648384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572913.627882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572913.627884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572913.627885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572913.627888:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9667 00000100:00080000:2.0:1788572913.627892:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572913.627895:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572913.627899:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572913.627903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572913.627912:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569648512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572913.627920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572913.627922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572913.627924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572913.627927:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9667 00000100:00080000:2.0:1788572913.627930:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572913.627932:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572913.627936:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572913.627939:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572913/real 1788572913] req@ffff920a60ac4700 x1875444569648128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572913.627946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569648640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572913.627964:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569648128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572913.627975:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572913.627979:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572913.627982:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569648128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572913.628003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648256:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572913.628019:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572913.628027:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572913/real 1788572913] req@ffff920a60ac7800 x1875444569648256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572913.628038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569648256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572913.628047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572913.628050:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572913.628054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648256:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572913.628060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572913.628082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572913.628093:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572913/real 1788572913] req@ffff920a60ac5500 x1875444569648384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572913.628102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569648384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572913.628108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572913.628110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572913.628113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572913.628117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648512:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572913.628131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572913.628138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572913/real 1788572913] req@ffff920a60ac6300 x1875444569648512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572913.628146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569648512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572913.628151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572913.628153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572913.628155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648512:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572913.628159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572913.628168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572913.628173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572913/real 1788572913] req@ffff920a60ac7100 x1875444569648640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572913.628180:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569648640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572913.628185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572913.628187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572913.628189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648640:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572914.075763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572914.651705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572915.099756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572915.675683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572916.123704:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572916.699675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572917.147713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572917.723713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572918.171760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572918.747754:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572918.747764:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572918.747769:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572918.747779:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9672 00000100:00100000:3.0:1788572918.747779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572918.747793:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572918.747801:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572918.747835:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569648768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572918.747858:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572918.747861:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572918.747863:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572918.747866:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9672 00000100:00080000:1.0:1788572918.747868:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572918.747871:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572918.747881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572918.747890:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569648896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572918.747896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569648768:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572918.747900:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572918.747906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572918.747907:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572918.747912:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9672 00000100:00080000:1.0:1788572918.747916:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572918.747921:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572918.747927:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572918.747941:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569649024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572918.747951:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572918.747955:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788572918.747960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572918.747961:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572918.747968:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9672 00000100:00080000:1.0:1788572918.747974:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572918.747977:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572918.747983:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572918.747996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572918/real 1788572918] req@ffff920a52891180 x1875444569648768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572918.747997:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569649152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572918.748028:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572918.748034:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572918.748036:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572918.748040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569648768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572918.748045:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9672 00000100:00080000:0.0:1788572918.748053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572918.748054:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572918.748060:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788572918.748063:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572918.748067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569648768:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572918.748074:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572918.748087:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569649280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572918.748092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648896:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572918.748124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572918.748140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572918/real 1788572918] req@ffff920a52890e00 x1875444569648896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572918.748150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569648896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572918.748156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572918.748159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572918.748161:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569648896:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572918.748168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572918.748179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572918.748184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572918/real 1788572918] req@ffff920a52893480 x1875444569649024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572918.748192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569649024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572918.748201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572918.748203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572918.748205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572918.748209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649152:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572918.748219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572918.748224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572918/real 1788572918] req@ffff920a52891f80 x1875444569649152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572918.748231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569649152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572918.748237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572918.748239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572918.748241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649152:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572918.748259:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572918.748270:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572918.748276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572918/real 1788572918] req@ffff920a52892a00 x1875444569649280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572918.748283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569649280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572943 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572918.748288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572918.748289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572918.748292:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649280:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572919.195832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572919.771768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572920.219808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572920.795740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572921.243770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572921.819698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572922.267740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572922.843717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572923.291727:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572923.867715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572923.867743:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572923.867750:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572923.867755:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572923.867765:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9677 00000100:00080000:2.0:1788572923.867775:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572923.867784:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572923.867821:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569649408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572923.867850:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572923.867855:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572923.867857:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572923.867862:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9677 00000100:00080000:2.0:1788572923.867867:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572923.867872:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572923.867883:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572923.867897:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569649536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572923.867909:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572923.867912:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569649408:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572923.867914:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572923.867920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572923.867927:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9677 00000100:00080000:2.0:1788572923.867931:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572923.867935:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572923.867941:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572923.867957:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569649664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572923.867968:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572923.867972:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572923.867974:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572923.867979:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9677 00000100:00080000:2.0:1788572923.867984:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572923.867990:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788572923.867991:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572923.868001:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572923.868014:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569649792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572923.868025:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572923.868029:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572923.868031:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788572923.868037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572923/real 1788572923] req@ffff920a60ac6a00 x1875444569649408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572923.868038:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9677 00000100:00080000:2.0:1788572923.868046:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572923.868054:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572923.868061:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572923.868076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569649408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572923.868076:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569649920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572923.868111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572923.868117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572923.868123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569649408:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572923.868148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649536:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572923.868174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572923.868189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572923/real 1788572923] req@ffff920a60ac5180 x1875444569649536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572923.868203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569649536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572923.868212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572923.868216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572923.868220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649536:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572923.868227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572923.868245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572923.868256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572923/real 1788572923] req@ffff920a60ac7100 x1875444569649664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572923.868268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569649664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572923.868277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572923.868280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572923.868284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572923.868290:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649792:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572923.868312:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572923.868325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572923/real 1788572923] req@ffff920a60ac6300 x1875444569649792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572923.868338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569649792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572923.868346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572923.868350:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572923.868353:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572923.868360:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572923.868382:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572923.868395:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572923/real 1788572923] req@ffff920a60ac5500 x1875444569649920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572923.868442:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569649920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572948 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572923.868464:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572923.868468:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572923.868471:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569649920:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572924.315763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572924.891697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572925.339728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572925.915689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572926.363756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572926.939675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572927.387796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572927.963680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572928.411744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572928.987668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572928.987713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572928.987718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572928.987722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572928.987730:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9682 00000100:00080000:1.0:1788572928.987740:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572928.987748:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572928.987782:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569650048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572928.987801:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572928.987804:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572928.987805:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572928.987808:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9682 00000100:00080000:1.0:1788572928.987811:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572928.987814:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572928.987821:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572928.987830:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569650176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572928.987838:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572928.987841:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572928.987842:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572928.987848:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9682 00000100:00100000:0.0:1788572928.987850:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569650048:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572928.987851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572928.987855:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572928.987862:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572928.987870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569650304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572928.987877:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572928.987879:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572928.987881:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572928.987884:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9682 00000100:00080000:1.0:1788572928.987886:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572928.987888:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572928.987892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572928.987898:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569650432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572928.987904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572928.987907:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572928.987908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572928.987911:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9682 00000100:00080000:1.0:1788572928.987915:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788572928.987916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572928.987922:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572928.987926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572928.987932:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569650560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572928.987960:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572928/real 1788572928] req@ffff920a52890a80 x1875444569650048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572928.987974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569650048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572928.987982:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572928.987986:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572928.987988:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569650048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572928.987993:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650176:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572928.988009:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572928.988019:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572928/real 1788572928] req@ffff920a52892680 x1875444569650176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572928.988032:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569650176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572928.988040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572928.988044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572928.988048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572928.988081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572928.988098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572928.988107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572928/real 1788572928] req@ffff920a52891500 x1875444569650304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572928.988115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569650304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572928.988120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572928.988123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572928.988125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572928.988129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650432:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572928.988139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572928.988145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572928/real 1788572928] req@ffff920a52893800 x1875444569650432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572928.988152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569650432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572928.988157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572928.988159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572928.988161:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572928.988164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572928.988173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572928.988179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572928/real 1788572928] req@ffff920a52891c00 x1875444569650560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572928.988186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569650560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572953 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572928.988191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572928.988193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572928.988194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650560:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572929.435712:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572930.011672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572930.459783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572931.035721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572931.483855:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572932.059699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572932.507786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572933.083688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572933.531742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572934.107672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572934.107706:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572934.107713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572934.107717:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572934.107727:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9687 00000100:00080000:2.0:1788572934.107738:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572934.107746:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572934.107784:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569650688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572934.107814:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572934.107819:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572934.107821:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572934.107826:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9687 00000100:00080000:2.0:1788572934.107831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572934.107837:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572934.107848:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572934.107855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569650688:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572934.107866:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569650816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572934.107879:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572934.107884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572934.107886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572934.107892:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9687 00000100:00080000:2.0:1788572934.107896:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572934.107901:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572934.107907:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788572934.107912:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572934.107922:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569650944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572934.107945:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788572934.107950:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff920a60ac7800 x1875444569650688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.107956:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572934.107961:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572934.107971:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9687 00000100:00080000:2.0:1788572934.107976:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788572934.107979:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569650688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572934.107984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572934.107995:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788572934.107999:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572934.108007:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572934.108013:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569651072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572934.108014:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569650688:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572934.108032:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572934.108036:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572934.108039:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572934.108045:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9687 00000100:00100000:0.0:1788572934.108047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650816:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788572934.108051:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572934.108058:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572934.108065:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572934.108073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572934.108082:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569651200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572934.108086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff920a60ac4700 x1875444569650816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572934.108113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569650816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572934.108125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572934.108128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572934.108131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572934.108135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572934.108147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572934.108154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff920a60ac4380 x1875444569650944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572934.108161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569650944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572934.108175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572934.108178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572934.108182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569650944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572934.108187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651072:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572934.108199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572934.108206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff920a60ac4000 x1875444569651072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572934.108214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569651072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572934.108218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572934.108220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572934.108223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572934.108234:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572934.108245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572934.108250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572934/real 1788572934] req@ffff920a60ac4a80 x1875444569651200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572934.108257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569651200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572934.108262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572934.108264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572934.108266:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651200:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572934.555696:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572935.131686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572935.579800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572936.155663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572936.603733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572937.179743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572937.627842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572938.203672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572938.651781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572939.227738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572939.227767:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.227779:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572939.227784:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.227793:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9692 00000100:00080000:1.0:1788572939.227803:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.227810:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572939.227846:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569651328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.227869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.227872:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572939.227873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.227876:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9692 00000100:00080000:1.0:1788572939.227879:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.227889:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572939.227900:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572939.227916:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569651456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572939.227930:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569651328:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572939.227942:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.227947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572939.227950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.227956:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9692 00000100:00080000:1.0:1788572939.227960:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.227966:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572939.227971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572939.227983:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569651584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.227992:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.227994:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572939.227996:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.227998:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9692 00000100:00080000:1.0:1788572939.228014:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.228019:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572939.228025:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788572939.228032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572939.228047:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569651712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572939.228068:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572939.228073:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572939.228075:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572939.228082:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9692 00000100:00100000:0.0:1788572939.228085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff920a52890380 x1875444569651328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572939.228089:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572939.228102:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572939.228112:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572939.228126:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569651840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572939.228131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569651328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572939.228189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572939.228197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572939.228202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569651328:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572939.228235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651456:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572939.228264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572939.228280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff920a52893b80 x1875444569651456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572939.228295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569651456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572939.228305:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572939.228309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572939.228313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572939.228321:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572939.228341:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572939.228352:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff920a52890000 x1875444569651584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572939.228366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569651584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572939.228376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572939.228380:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572939.228384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572939.228391:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651712:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572939.228413:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572939.228425:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff920a52891c00 x1875444569651712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572939.228452:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569651712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572939.228463:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572939.228466:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572939.228470:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572939.228477:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572939.228497:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572939.228518:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572939/real 1788572939] req@ffff920a52893800 x1875444569651840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572939.228531:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569651840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572939.228540:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572939.228544:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572939.228547:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569651840:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572939.675776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572940.251758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572940.699813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572941.275706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572941.723741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572942.299686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572942.747765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572943.323696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572943.771758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572944.347674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572944.347703:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572944.347711:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572944.347715:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572944.347726:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9697 00000100:00080000:1.0:1788572944.347739:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572944.347748:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572944.347787:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569651968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572944.347820:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572944.347825:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572944.347828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572944.347834:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9697 00000100:00080000:1.0:1788572944.347840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572944.347845:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572944.347852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569651968:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572944.347866:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572944.347887:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569652096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572944.347901:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572944.347905:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572944.347907:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572944.347911:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9697 00000100:00080000:1.0:1788572944.347916:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572944.347921:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572944.347927:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572944.347943:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569652224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572944.347956:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652096:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788572944.347958:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572944.347966:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572944.347968:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572944.347974:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9697 00000100:00080000:1.0:1788572944.347979:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572944.347984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572944.347991:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572944.348005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572944.348006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569652352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572944.348028:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572944.348030:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572944.348032:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572944.348035:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9697 00000100:00080000:1.0:1788572944.348039:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572944.348042:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572944.348046:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572944.348049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572944.348057:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569652480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572944.348097:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572944.795805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572945.371694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572945.819739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572946.395694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572946.843735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572947.419685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788572947.419863:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788572947.419906:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788572947.419934:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572947.419974:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572947.419985:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572947.419996:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788572947.420010:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788572947.420013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572947] req@ffff920a52890a80 x1875444569652224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572947.420041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569652224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572947.420052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572947.420058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572947.420062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572947.420071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572947] req@ffff920a52892680 x1875444569652096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572947.420085:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569652096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572947.420093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572947.420098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572947.420101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652096:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572947.420108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572947] req@ffff920a52891500 x1875444569651968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572947.420122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569651968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572947.420131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572947.420134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572947.420138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569651968:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572947.420161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572947] req@ffff920a52891f80 x1875444569652480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572947.420175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569652480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572947.420182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572947.420185:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572947.420187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572947.420193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572944/real 1788572947] req@ffff920a52892a00 x1875444569652352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572947.420205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569652352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572947.420213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572947.420217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572947.420219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652352:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572947.867740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572948.443686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572948.891774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788572949.467739:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572949.467745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572949.467755:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572949.467760:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.467773:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9702 00000100:00080000:1.0:1788572949.467788:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.467798:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572949.467849:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569652608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.467886:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.467891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572949.467893:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.467900:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9702 00000100:00080000:1.0:1788572949.467914:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.467921:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572949.467934:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569652608:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788572949.467934:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572949.467955:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569652736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.467965:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.467970:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572949.467972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.467977:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9702 00000100:00080000:1.0:1788572949.467981:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.467992:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572949.467998:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572949.468015:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569652864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.468026:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.468030:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572949.468032:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788572949.468038:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572949.468039:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9702 00000100:00080000:1.0:1788572949.468047:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.468052:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572949.468059:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572949.468075:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569652992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572949.468085:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572949.468091:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788572949.468092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff920a52893480 x1875444569652608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572949.468094:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572949.468103:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9702 00000100:00080000:1.0:1788572949.468110:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572949.468117:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572949.468122:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572949.468134:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569653120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788572949.468142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569652608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572949.468157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572949.468163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572949.468168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569652608:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572949.468203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652736:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572949.468246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572949.468269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff920a52890e00 x1875444569652736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572949.468282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569652736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572949.468297:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572949.468301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572949.468304:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652736:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572949.468310:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788572949.468332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572949.468344:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff920a52891180 x1875444569652864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572949.468358:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569652864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572949.468367:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572949.468370:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572949.468374:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572949.468379:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652992:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572949.468400:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572949.468410:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff920a52890700 x1875444569652992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572949.468421:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569652992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572949.468429:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572949.468433:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572949.468437:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569652992:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788572949.468443:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788572949.468464:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788572949.468475:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572949/real 1788572949] req@ffff920a52893100 x1875444569653120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572949.468486:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569653120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572949.468494:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572949.468497:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788572949.468500:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653120:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572949.915725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572950.491686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572950.939781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572951.515684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572951.963722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572952.539692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572952.987741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572953.563675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572954.011738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572954.587675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572954.587736:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572954.587744:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572954.587748:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572954.587757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9707 00000100:00080000:0.0:1788572954.587767:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572954.587775:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572954.587813:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569653248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572954.587838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572954.587842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572954.587843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572954.587846:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9707 00000100:00080000:0.0:1788572954.587850:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572954.587854:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572954.587862:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572954.587873:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569653376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572954.587881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569653248:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572954.587883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572954.587890:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572954.587892:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572954.587896:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9707 00000100:00080000:0.0:1788572954.587898:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572954.587900:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572954.587904:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572954.587913:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569653504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572954.587920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572954.587922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572954.587923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572954.587926:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9707 00000100:00080000:0.0:1788572954.587930:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572954.587932:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572954.587935:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572954.587944:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569653632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572954.587952:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788572954.587953:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572954.587957:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572954.587961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572954.587965:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9707 00000100:00080000:0.0:1788572954.587968:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572954.587970:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572954.587974:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572954.587982:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569653760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572954.587995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff920a48353b80 x1875444569653248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572954.588010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569653248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572954.588021:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572954.588025:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572954.588030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569653248:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572954.588053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653376:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572954.588078:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572954.588102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff920a48353800 x1875444569653376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572954.588116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569653376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572954.588124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572954.588127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572954.588131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653376:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572954.588137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572954.588156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572954.588166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff920a48351500 x1875444569653504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572954.588177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569653504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572954.588186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572954.588189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572954.588192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572954.588198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653632:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572954.588215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572954.588224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff920a48351c00 x1875444569653632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572954.588242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569653632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572954.588250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572954.588253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572954.588256:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653632:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572954.588263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572954.588279:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572954.588288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572954/real 1788572954] req@ffff920a48350a80 x1875444569653760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572954.588301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569653760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572954.588310:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572954.588313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572954.588317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569653760:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572955.035774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572955.611660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572956.059769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572956.635659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572957.083772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572957.659721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572958.107820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572958.683712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572959.131800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572959.707697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572959.707724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572959.707734:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572959.707739:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572959.707752:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9713 00000100:00080000:1.0:1788572959.707764:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572959.707772:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572959.707820:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569653888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572959.707845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572959.707848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572959.707850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572959.707853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9713 00000100:00080000:1.0:1788572959.707857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572959.707861:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572959.707874:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572959.707885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569653888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572959.707886:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569654016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572959.707911:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572959.707914:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572959.707916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572959.707922:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9713 00000100:00080000:1.0:1788572959.707923:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572959.707926:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572959.707930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572959.707941:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569654144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572959.707948:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572959.707951:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572959.707953:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572959.707956:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9713 00000100:00080000:1.0:1788572959.707959:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572959.707962:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572959.707965:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788572959.707971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572959.707977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569654272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572959.707988:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572959.707991:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572959.707992:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572959.707996:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9713 00000100:00080000:1.0:1788572959.707999:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572959.708004:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572959.708010:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572959.708016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572959/real 1788572959] req@ffff920a52892300 x1875444569653888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572959.708021:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569654400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572959.708090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569653888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572959.708101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572959.708104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572959.708108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569653888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572959.708127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654016:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572959.708144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572959.708153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572959/real 1788572959] req@ffff920a52892d80 x1875444569654016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572959.708162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569654016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572959.708169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572959.708172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572959.708176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654016:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572959.708183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572959.708203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572959.708215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572959/real 1788572959] req@ffff920a52891880 x1875444569654144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572959.708227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569654144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572959.708234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572959.708237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572959.708239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572959.708244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654272:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572959.708260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572959.708267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572959/real 1788572959] req@ffff920a52893100 x1875444569654272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572959.708274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569654272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572959.708280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572959.708282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572959.708283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654272:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572959.708287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572959.708299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572959.708305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572959/real 1788572959] req@ffff920a52890700 x1875444569654400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572959.708312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569654400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572984 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572959.708317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572959.708319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572959.708321:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654400:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572960.155736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572960.731677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572961.179757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572961.755677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572962.203726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572962.779771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572963.227807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572963.803700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572964.251765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788572964.827706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572964.827743:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572964.827753:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572964.827757:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572964.827768:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9718 00000100:00080000:1.0:1788572964.827780:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572964.827787:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572964.827840:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569654528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572964.827875:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572964.827879:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572964.827880:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572964.827885:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9718 00000100:00080000:1.0:1788572964.827890:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572964.827894:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572964.827905:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572964.827913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569654528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572964.827920:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569654656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572964.827931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572964.827935:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572964.827937:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572964.827944:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9718 00000100:00080000:1.0:1788572964.827948:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572964.827953:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572964.827959:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572964.827977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569654784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572964.827990:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572964.828002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572964.828006:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572964.828009:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572964.828014:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9718 00000100:00080000:1.0:1788572964.828019:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572964.828023:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572964.828029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572964/real 1788572964] req@ffff920a52891180 x1875444569654528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788572964.828030:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572964.828052:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569654912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572964.828058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569654528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572964.828081:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572964.828084:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572964.828085:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572964.828090:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9718 00000100:00080000:1.0:1788572964.828095:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572964.828106:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572964.828115:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788572964.828115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572964.828125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572964.828132:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569655040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572964.828134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569654528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572964.828176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654656:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572964.828196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572964.828205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572964/real 1788572964] req@ffff920a52890e00 x1875444569654656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572964.828213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569654656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572964.828218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572964.828221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572964.828223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654656:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572964.828227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572964.828237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572964.828243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572964/real 1788572964] req@ffff920a52893480 x1875444569654784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572964.828255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569654784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572964.828260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572964.828262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572964.828264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572964.828268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654912:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572964.828278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572964.828283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572964/real 1788572964] req@ffff920a52892a00 x1875444569654912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572964.828289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569654912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572964.828294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572964.828296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572964.828298:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569654912:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572964.828301:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572964.828311:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572964.828317:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572964/real 1788572964] req@ffff920a52891f80 x1875444569655040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572964.828323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569655040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572989 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572964.828332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572964.828334:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572964.828336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655040:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572965.275732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572965.852040:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572966.299700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572966.875703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572967.323775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572967.899650:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572968.347704:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572968.923694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572969.371773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572969.947652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788572969.947702:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572969.947714:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788572969.947719:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572969.947730:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9723 00000100:00080000:1.0:1788572969.947740:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572969.947747:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572969.947779:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569655168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572969.947802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572969.947805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788572969.947806:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572969.947811:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9723 00000100:00080000:1.0:1788572969.947814:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572969.947817:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572969.947825:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572969.947835:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569655296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572969.947844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572969.947846:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569655168:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788572969.947848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788572969.947851:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572969.947858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9723 00000100:00080000:1.0:1788572969.947860:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572969.947863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572969.947866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572969.947875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569655424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572969.947881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572969.947884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788572969.947885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572969.947888:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9723 00000100:00080000:1.0:1788572969.947890:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572969.947893:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572969.947896:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572969.947902:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569655552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572969.947909:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572969.947911:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788572969.947913:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572969.947915:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9723 00000100:00080000:1.0:1788572969.947917:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788572969.947922:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788572969.947924:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788572969.947929:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572969.947936:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569655680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788572969.947966:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572969/real 1788572969] req@ffff920a52891500 x1875444569655168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572969.947987:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569655168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572969.947998:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572969.948002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572969.948006:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569655168:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788572969.948030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655296:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572969.948053:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572969.948066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572969/real 1788572969] req@ffff920a52892680 x1875444569655296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572969.948079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569655296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572969.948085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572969.948088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572969.948091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655296:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572969.948095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572969.948109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572969.948116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572969/real 1788572969] req@ffff920a52890a80 x1875444569655424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572969.948123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569655424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572969.948130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572969.948133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572969.948136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572969.948142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655552:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572969.948164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572969.948176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572969/real 1788572969] req@ffff920a52893800 x1875444569655552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572969.948189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569655552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572969.948197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572969.948200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572969.948204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655552:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572969.948210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572969.948230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572969.948241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572969/real 1788572969] req@ffff920a52891c00 x1875444569655680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572969.948253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569655680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788572994 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572969.948262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572969.948265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572969.948269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655680:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572970.395714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572970.971701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572971.419708:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572971.995744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572972.443787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572973.019693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572973.467728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572974.043698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572974.491770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572975.067700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572975.067754:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.067762:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572975.067767:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.067781:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9728 00000100:00080000:3.0:1788572975.067795:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.067805:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572975.067856:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569655808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.067900:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.067904:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572975.067906:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.067912:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9728 00000100:00080000:3.0:1788572975.067919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.067925:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572975.067937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572975.067952:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569655808:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788572975.067957:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569655936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.067990:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.067994:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572975.067997:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.068004:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9728 00000100:00080000:3.0:1788572975.068009:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.068015:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572975.068021:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572975.068050:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569656064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572975.068054:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572975.068068:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572975.068072:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572975.068074:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572975.068080:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9728 00000100:00080000:3.0:1788572975.068084:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.068088:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572975.068094:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572975.068108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff920a529b3b80 x1875444569655808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572975.068112:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569656192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572975.068141:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788572975.068142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569655808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572975.068162:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572975.068167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572975.068167:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572975.068182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788572975.068183:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9728 00000100:00100000:2.0:1788572975.068191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569655808:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572975.068194:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572975.068205:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572975.068213:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572975.068220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655936:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572975.068233:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569656320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788572975.068246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572975.068263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff920a529b0380 x1875444569655936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572975.068275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569655936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572975.068281:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572975.068284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572975.068286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569655936:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572975.068291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788572975.068303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572975.068309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff920a529b1180 x1875444569656064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572975.068317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569656064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572975.068326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572975.068329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572975.068333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788572975.068339:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656192:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572975.068361:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572975.068373:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff920a529b1500 x1875444569656192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572975.068386:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569656192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572975.068394:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572975.068398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572975.068401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656192:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788572975.068432:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788572975.068456:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788572975.068489:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572975/real 1788572975] req@ffff920a529b3b80 x1875444569656320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572975.068504:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569656320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788572975.068513:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788572975.068517:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788572975.068520:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656320:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572975.515744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572976.091707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572976.539772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572977.115700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572977.563769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572978.139684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572978.587777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572979.163698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572979.611788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572980.187740:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572980.187752:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788572980.187755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572980.187757:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572980.187774:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9733 00000100:00080000:0.0:1788572980.187786:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572980.187799:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572980.187844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569656448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572980.187881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572980.187887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572980.187891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572980.187897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9733 00000100:00080000:0.0:1788572980.187903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572980.187909:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572980.187922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572980.187922:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569656448:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572980.187949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569656576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572980.187967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572980.187972:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572980.187975:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572980.187982:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9733 00000100:00080000:0.0:1788572980.188005:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572980.188019:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788572980.188020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788572980.188032:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572980.188057:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569656704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572980.188074:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572980.188077:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788572980.188080:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff920a48350700 x1875444569656448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572980.188081:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572980.188098:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9733 00000100:00080000:0.0:1788572980.188107:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572980.188114:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572980.188120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569656448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788572980.188121:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572980.188142:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569656832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788572980.188144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572980.188159:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572980.188163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788572980.188165:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572980.188171:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572980.188181:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9733 00000100:00100000:1.0:1788572980.188186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569656448:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572980.188187:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572980.188199:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572980.188204:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788572980.188212:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569656960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572980.188244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656576:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572980.188274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572980.188291:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff920a48350380 x1875444569656576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572980.188304:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569656576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572980.188313:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572980.188318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572980.188322:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656576:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572980.188330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572980.188357:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572980.188375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff920a48353480 x1875444569656704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572980.188389:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569656704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572980.188398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572980.188406:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572980.188425:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572980.188433:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656832:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572980.188476:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572980.188492:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff920a48350a80 x1875444569656832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572980.188506:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569656832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572980.188514:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572980.188518:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572980.188521:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656832:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572980.188528:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572980.188552:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572980.188565:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572980/real 1788572980] req@ffff920a48351c00 x1875444569656960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572980.188578:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569656960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572980.188642:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572980.188646:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572980.188651:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569656960:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572980.635756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572981.211718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572981.659768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572982.235706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572982.683754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572983.259679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572983.707747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572984.283699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572984.731749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788572985.307685:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788572985.307685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788572985.307695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788572985.307699:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.307707:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9738 00000100:00080000:0.0:1788572985.307715:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.307722:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788572985.307752:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569657088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.307774:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.307777:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788572985.307778:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.307781:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9738 00000100:00080000:0.0:1788572985.307784:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.307789:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572985.307799:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788572985.307812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569657088:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788572985.307815:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569657216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788572985.307831:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.307835:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788572985.307837:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.307841:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9738 00000100:00080000:0.0:1788572985.307846:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.307849:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572985.307854:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788572985.307866:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569657344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572985.307877:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788572985.307879:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788572985.307884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788572985.307886:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788572985.307892:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9738 00000100:00080000:0.0:1788572985.307896:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.307901:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572985.307907:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572985.307909:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff920a48351500 x1875444569657088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788572985.307926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569657472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572985.307933:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569657088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788572985.307942:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788572985.307946:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788572985.307947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788572985.307952:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788572985.307952:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788572985.307962:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9738 00000100:00100000:1.0:1788572985.307963:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569657088:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788572985.307971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788572985.307977:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788572985.307982:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572985.307990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657216:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788572985.307998:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569657600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572985.308011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572985.308023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff920a48353800 x1875444569657216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572985.308034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569657216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572985.308040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572985.308043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572985.308047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657216:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572985.308052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572985.308080:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572985.308092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff920a48353b80 x1875444569657344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572985.308106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569657344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572985.308121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572985.308124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572985.308127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572985.308134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657472:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572985.308156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572985.308167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff920a48351c00 x1875444569657472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572985.308179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569657472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572985.308188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572985.308191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572985.308195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657472:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572985.308211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572985.308233:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572985.308244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572985/real 1788572985] req@ffff920a48350a80 x1875444569657600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572985.308256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569657600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572985.308265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572985.308268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572985.308272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657600:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572985.755738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572986.331708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572986.779816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572987.355677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572987.803728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572988.379716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572988.827745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572989.403661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572989.851776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572990.427673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788572990.427687:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572990.427693:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788572990.427696:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572990.427708:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9743 00000100:00080000:2.0:1788572990.427718:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572990.427725:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788572990.427757:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569657728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572990.427776:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572990.427779:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788572990.427780:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572990.427783:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9743 00000100:00080000:2.0:1788572990.427786:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572990.427788:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572990.427796:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572990.427805:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569657856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572990.427814:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572990.427816:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569657728:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788572990.427817:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788572990.427821:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572990.427828:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9743 00000100:00080000:2.0:1788572990.427830:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572990.427832:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572990.427836:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788572990.427847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569657984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788572990.427854:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572990.427857:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788572990.427859:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572990.427862:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9743 00000100:00080000:2.0:1788572990.427865:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572990.427867:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572990.427871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788572990.427878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569658112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572990.427883:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788572990.427890:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788572990.427893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788572990.427894:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788572990.427899:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9743 00000100:00080000:2.0:1788572990.427905:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788572990.427909:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788572990.427915:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572990.427918:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff920a60ac5c00 x1875444569657728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788572990.427930:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569658240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788572990.427936:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569657728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572990.427958:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572990.427962:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572990.427966:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569657728:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788572990.427984:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657856:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572990.428000:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572990.428009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff920a60ac4e00 x1875444569657856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572990.428018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569657856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572990.428024:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572990.428026:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572990.428029:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657856:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572990.428035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572990.428046:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572990.428052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff920a60ac4a80 x1875444569657984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572990.428060:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569657984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572990.428065:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572990.428067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572990.428069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569657984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572990.428072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658112:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572990.428083:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572990.428089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff920a60ac4000 x1875444569658112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572990.428102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569658112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572990.428110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572990.428114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572990.428117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658112:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572990.428124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572990.428145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572990.428156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572990/real 1788572990] req@ffff920a60ac4380 x1875444569658240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572990.428165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569658240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572990.428180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572990.428182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572990.428184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658240:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788572990.875715:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572991.451714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572991.899799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572992.475701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572992.923719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572993.499695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572993.947793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572994.523684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572994.971785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572995.547736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788572995.547772:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572995.547781:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788572995.547786:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572995.547798:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9748 00000100:00080000:3.0:1788572995.547811:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572995.547819:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788572995.547868:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569658368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572995.547901:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572995.547906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788572995.547908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572995.547914:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9748 00000100:00080000:3.0:1788572995.547918:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572995.547924:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788572995.547932:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569658368:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788572995.547937:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572995.547953:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569658496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572995.547965:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572995.547969:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788572995.547972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572995.547977:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9748 00000100:00080000:3.0:1788572995.547982:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572995.547986:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572995.547992:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788572995.548006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569658624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572995.548017:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788572995.548018:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788572995.548023:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788572995.548025:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572995.548030:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9748 00000100:00080000:3.0:1788572995.548036:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788572995.548041:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572995.548047:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572995.548059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572995] req@ffff920a529b0e00 x1875444569658368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788572995.548062:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569658752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788572995.548086:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788572995.548091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569658368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788572995.548092:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788572995.548095:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788572995.548105:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9748 00000100:00080000:1.0:1788572995.548110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788572995.548110:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788572995.548118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572995.548124:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569658368:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788572995.548130:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788572995.548138:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788572995.548148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658496:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788572995.548154:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569658880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788572995.548176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572995.548188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572995] req@ffff920a529b1f80 x1875444569658496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572995.548197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569658496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572995.548203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572995.548205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572995.548207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658496:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572995.548211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788572995.548222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572995.548228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572995] req@ffff920a529b0000 x1875444569658624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572995.548235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569658624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572995.548240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572995.548242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572995.548244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788572995.548247:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658752:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572995.548259:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572995.548265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572995] req@ffff920a529b3100 x1875444569658752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572995.548277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569658752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572995.548282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572995.548284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572995.548286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658752:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788572995.548295:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788572995.548306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788572995.548312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788572995/real 1788572995] req@ffff920a529b2d80 x1875444569658880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788572995.548319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569658880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788572995.548324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788572995.548326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788572995.548328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569658880:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788572995.995828:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788572996.571748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572997.019852:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572997.595729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788572998.043785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572998.619740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788572999.067805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788572999.643691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573000.091788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573000.667703:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573000.667710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573000.667713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573000.667719:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573000.667730:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9754 00000100:00080000:3.0:1788573000.667740:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573000.667749:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573000.667789:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569659008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573000.667813:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573000.667817:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573000.667819:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573000.667824:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9754 00000100:00080000:3.0:1788573000.667828:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573000.667833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573000.667844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573000.667848:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569659008:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573000.667860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569659136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573000.667882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573000.667887:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573000.667889:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573000.667895:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9754 00000100:00080000:3.0:1788573000.667899:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573000.667904:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573000.667910:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573000.667915:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573000.667925:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569659264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573000.667937:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573000.667940:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573000.667943:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573000.667947:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9754 00000100:00080000:3.0:1788573000.667952:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573000.667957:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573000.667959:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573000/real 1788573000] req@ffff920a529b3480 x1875444569659008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573000.667964:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573000.667980:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569659392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573000.667983:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569659008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573000.668005:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573000.668010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573000.668010:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573000.668013:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573000.668020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788573000.668021:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9754 00000100:00080000:3.0:1788573000.668028:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573000.668033:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569659008:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573000.668037:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573000.668045:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573000.668058:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569659520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573000.668064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659136:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573000.668094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573000.668105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573000/real 1788573000] req@ffff920a529b0700 x1875444569659136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573000.668114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569659136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573000.668121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573000.668124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573000.668126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659136:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573000.668130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573000.668152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573000.668159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573000/real 1788573000] req@ffff920a529b1c00 x1875444569659264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573000.668168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569659264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573000.668178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573000.668180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573000.668182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573000.668186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659392:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573000.668198:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573000.668203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573000/real 1788573000] req@ffff920a529b2300 x1875444569659392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573000.668211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569659392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573000.668216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573000.668218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573000.668220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659392:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573000.668232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573000.668251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573000.668263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573000/real 1788573000] req@ffff920a529b2a00 x1875444569659520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573000.668276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569659520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573000.668285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573000.668288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573000.668291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659520:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573001.115716:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573001.691705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573002.139789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573002.715686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573003.163739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573003.739695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573004.187749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573004.763668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573005.211757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573005.787689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573005.787698:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573005.787704:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573005.787708:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573005.787717:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9759 00000100:00080000:0.0:1788573005.787726:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573005.787734:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573005.787772:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569659648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573005.787800:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573005.787804:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573005.787806:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573005.787812:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9759 00000100:00080000:0.0:1788573005.787817:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573005.787823:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573005.787828:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569659648:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573005.787835:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573005.787854:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569659776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573005.787868:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573005.787871:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573005.787873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573005.787876:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9759 00000100:00080000:0.0:1788573005.787880:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573005.787885:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573005.787891:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573005.787898:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573005.787914:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569659904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573005.787920:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573005/real 1788573005] req@ffff920a48353480 x1875444569659648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573005.787932:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573005.787941:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573005.787944:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573005.787953:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569659648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573005.787957:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9759 00000100:00080000:0.0:1788573005.787963:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573005.787970:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573005.787979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573005.787979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573005.787989:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573005.787995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569660032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573005.787997:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569659648:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573005.788012:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573005.788015:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573005.788016:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573005.788019:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9759 00000100:00080000:0.0:1788573005.788023:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573005.788025:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573005.788029:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573005.788035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659776:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573005.788039:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569660160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573005.788071:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573005.788087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573005/real 1788573005] req@ffff920a48350380 x1875444569659776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573005.788102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569659776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573005.788122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573005.788127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573005.788131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659776:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573005.788138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573005.788161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573005.788174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573005/real 1788573005] req@ffff920a48350700 x1875444569659904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573005.788187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569659904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573005.788195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573005.788199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573005.788202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569659904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573005.788209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660032:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573005.788232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573005.788245:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573005/real 1788573005] req@ffff920a48350a80 x1875444569660032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573005.788257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569660032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573005.788266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573005.788269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573005.788273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660032:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573005.788290:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573005.788312:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573005.788337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573005/real 1788573005] req@ffff920a48351c00 x1875444569660160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573005.788350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569660160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573030 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573005.788359:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573005.788362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573005.788396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660160:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573006.235782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573006.811665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573007.259756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573007.835675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573008.283760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573008.859658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573009.307712:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573009.883691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573010.331735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573010.907710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573010.907724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573010.907733:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573010.907738:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573010.907748:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9764 00000100:00080000:0.0:1788573010.907759:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573010.907769:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573010.907817:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569660288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573010.907838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573010.907842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573010.907843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573010.907847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9764 00000100:00080000:0.0:1788573010.907850:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573010.907854:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573010.907862:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573010.907869:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569660288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573010.907875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569660416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573010.907884:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573010.907888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573010.907890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573010.907892:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9764 00000100:00080000:0.0:1788573010.907894:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573010.907897:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573010.907900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573010.907908:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569660544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573010.907915:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573010.907917:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573010.907918:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573010.907920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9764 00000100:00080000:0.0:1788573010.907923:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573010.907926:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573010.907930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573010.907937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569660672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573010.907944:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573010.907947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573010.907948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573010.907950:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9764 00000100:00080000:0.0:1788573010.907952:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573010.907955:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573010.907959:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660416:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788573010.907961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573010.907974:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569660800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573010.907990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573010.908014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660672:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573010.908032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660800:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573011.355770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573011.931673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573012.379732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573012.955698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573013.403755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573013.979727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573013.979867:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573013.979910:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573013.979927:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573013.979975:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573013.979992:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573013.980003:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573010/real 1788573013] req@ffff920a48353800 x1875444569660416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788573013.980017:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573013.980039:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573013.980070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569660416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.980087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.980094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573013.980099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660416:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573013.980109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573010/real 1788573013] req@ffff920a48351500 x1875444569660544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.980120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569660544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.980127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.980130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573013.980133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573013.980140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573010/real 1788573013] req@ffff920a48351c00 x1875444569660672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.980155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569660672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.980163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.980167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573013.980172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660672:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573013.980180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573010/real 1788573013] req@ffff920a48353b80 x1875444569660288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.980195:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569660288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.980204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.980208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573013.980212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569660288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573013.980265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573010/real 1788573013] req@ffff920a48350a80 x1875444569660800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573013.980278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569660800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573035 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573013.980287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573013.980290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573013.980293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569660800:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573014.427713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573015.003686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573015.451776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573016.027690:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573016.027705:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573016.027710:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573016.027725:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9769 00000100:00080000:0.0:1788573016.027740:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573016.027748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573016.027757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788573016.027789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569660928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573016.027826:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573016.027831:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573016.027834:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573016.027841:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9769 00000100:00080000:0.0:1788573016.027847:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573016.027852:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573016.027864:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573016.027864:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569660928:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573016.027885:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569661056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573016.027896:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573016.027900:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573016.027902:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573016.027907:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9769 00000100:00080000:0.0:1788573016.027912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573016.027917:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573016.027939:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573016.027958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569661184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573016.027958:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573016.027979:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573016.027984:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573016.027986:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573016.027992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9769 00000100:00080000:0.0:1788573016.027997:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573016.028005:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573016.028007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff920a48350700 x1875444569660928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573016.028011:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573016.028033:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569661312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573016.028038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569660928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573016.028055:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573016.028062:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573016.028062:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573016.028064:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573016.028069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573016.028074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569660928:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573016.028089:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9769 00000100:00080000:0.0:1788573016.028094:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573016.028098:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573016.028105:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573016.028108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661056:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573016.028122:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569661440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573016.028150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573016.028168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff920a48350380 x1875444569661056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.028194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569661056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.028204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.028208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573016.028211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661056:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573016.028218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573016.028238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573016.028249:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff920a48353480 x1875444569661184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.028261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569661184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.028271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.028275:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573016.028278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573016.028285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661312:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573016.028304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573016.028314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff920a48350a80 x1875444569661312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.028325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569661312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.028335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.028338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573016.028342:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661312:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573016.028356:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573016.028373:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573016.028383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573016/real 1788573016] req@ffff920a48353b80 x1875444569661440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573016.028394:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569661440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573016.028404:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573016.028407:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573016.028410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661440:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573016.475815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573017.051739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573017.499784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573018.075686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573018.523799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573019.099685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573019.547809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573020.123694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573020.571794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573021.147697:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573021.147705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573021.147707:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573021.147712:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573021.147723:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9774 00000100:00080000:2.0:1788573021.147732:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573021.147739:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573021.147773:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569661568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573021.147795:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573021.147798:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573021.147800:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573021.147803:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9774 00000100:00080000:2.0:1788573021.147806:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573021.147809:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573021.147818:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573021.147828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569661696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573021.147838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573021.147842:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569661568:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573021.147844:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573021.147846:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573021.147849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9774 00000100:00080000:2.0:1788573021.147857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573021.147864:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573021.147870:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573021.147890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569661824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573021.147901:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573021.147911:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573021.147913:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573021.147918:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9774 00000100:00080000:2.0:1788573021.147922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573021.147927:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573021.147933:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573021.147945:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573021.147952:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569661952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573021.147972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573021.147976:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573021.147978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573021.147990:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff920a60ac4700 x1875444569661568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573021.147996:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9774 00000100:00080000:2.0:1788573021.148011:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573021.148019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573021.148022:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569661568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573021.148028:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573021.148034:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.148040:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573021.148047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569661568:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573021.148049:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569662080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573021.148078:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661696:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573021.148120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573021.148131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff920a60ac7800 x1875444569661696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.148140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569661696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.148146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.148148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573021.148151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661696:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.148155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573021.148168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573021.148174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff920a60ac5500 x1875444569661824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.148182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569661824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.148187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.148189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573021.148191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573021.148194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661952:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573021.148207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573021.148213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff920a60ac6300 x1875444569661952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.148220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569661952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.148225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.148227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573021.148229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569661952:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573021.148232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573021.148242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573021.148248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573021/real 1788573021] req@ffff920a60ac7100 x1875444569662080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573021.148255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569662080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573021.148264:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573021.148267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573021.148269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662080:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573021.595846:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573022.171727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573022.619759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573023.195682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573023.643775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573024.219705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573024.667770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573025.243868:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573025.691756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573026.267660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573026.267669:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.267676:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573026.267680:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.267688:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9779 00000100:00080000:2.0:1788573026.267697:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.267703:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573026.267743:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569662208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573026.267765:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.267768:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573026.267769:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.267774:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9779 00000100:00080000:2.0:1788573026.267777:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.267780:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573026.267789:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573026.267795:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569662208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573026.267801:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569662336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573026.267817:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.267820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573026.267822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.267825:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9779 00000100:00080000:2.0:1788573026.267828:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.267830:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573026.267834:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573026.267844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569662464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573026.267850:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573026.267854:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788573026.267856:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573026.267857:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.267864:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9779 00000100:00080000:2.0:1788573026.267867:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.267871:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573026.267875:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573026.267883:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569662592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573026.267890:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573026.267891:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff920a60ac5180 x1875444569662208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573026.267893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573026.267897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573026.267904:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9779 00000100:00080000:2.0:1788573026.267909:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573026.267913:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573026.267916:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573026.267917:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569662208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573026.267926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569662720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573026.267928:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.267939:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573026.267944:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569662208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573026.267964:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662336:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573026.267980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573026.268004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff920a60ac6a00 x1875444569662336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.268015:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569662336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.268022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.268024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573026.268027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662336:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573026.268030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573026.268045:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573026.268052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff920a60ac7100 x1875444569662464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.268060:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569662464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.268065:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.268067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573026.268069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573026.268073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662592:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573026.268084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573026.268089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff920a60ac6300 x1875444569662592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.268097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569662592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.268101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.268103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573026.268106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662592:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573026.268109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573026.268118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573026.268124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573026/real 1788573026] req@ffff920a60ac5500 x1875444569662720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573026.268130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569662720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573026.268135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573026.268137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573026.268139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662720:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573026.715719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573027.291701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573027.739750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573028.315674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573028.763812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573029.339767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573029.787740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573030.363706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573030.811765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573031.387658:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.387667:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573031.387670:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573031.387675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573031.387680:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9784 00000100:00080000:2.0:1788573031.387690:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.387696:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573031.387730:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569662848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.387753:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.387755:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573031.387756:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.387759:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9784 00000100:00080000:2.0:1788573031.387762:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.387766:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573031.387774:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573031.387786:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569662848:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573031.387787:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569662976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.387800:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.387803:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573031.387804:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.387806:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9784 00000100:00080000:2.0:1788573031.387809:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.387811:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573031.387814:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573031.387823:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569663104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.387830:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.387832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573031.387833:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.387835:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9784 00000100:00080000:2.0:1788573031.387838:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.387841:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573031.387845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573031.387852:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573031.387854:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569663232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573031.387866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573031.387868:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573031.387870:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573031.387872:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9784 00000100:00080000:2.0:1788573031.387874:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573031.387876:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573031.387879:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573031.387884:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569663360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573031.387919:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff920a60ac7800 x1875444569662848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.387941:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569662848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.387953:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.387958:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573031.387963:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569662848:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573031.387985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662976:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573031.388004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573031.388015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff920a60ac4700 x1875444569662976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.388028:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569662976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.388036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.388041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573031.388044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569662976:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573031.388050:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573031.388071:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573031.388079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff920a60ac4380 x1875444569663104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.388087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569663104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.388093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.388095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573031.388097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573031.388100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663232:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573031.388113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573031.388119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff920a60ac4000 x1875444569663232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.388127:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569663232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.388132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.388133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573031.388135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663232:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573031.388138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573031.388148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573031.388153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573031/real 1788573031] req@ffff920a60ac4a80 x1875444569663360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573031.388160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569663360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573031.388165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573031.388167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573031.388168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663360:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573031.835752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573032.411694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573032.859765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573033.435678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573033.883774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573034.459689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573034.907766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573035.483686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573035.931742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573036.507692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573036.507709:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573036.507715:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573036.507718:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573036.507726:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9789 00000100:00080000:0.0:1788573036.507734:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573036.507740:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573036.507780:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569663488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573036.507822:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573036.507827:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573036.507829:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573036.507835:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9789 00000100:00080000:0.0:1788573036.507840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573036.507843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569663488:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573036.507849:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573036.507859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573036.507877:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569663616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573036.507889:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573036.507893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573036.507895:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573036.507899:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9789 00000100:00080000:0.0:1788573036.507904:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573036.507908:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573036.507913:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573036.507914:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573036.507928:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569663744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573036.507940:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573036.507944:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573036.507947:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573036.507951:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9789 00000100:00080000:0.0:1788573036.507955:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573036.507958:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff920a48351c00 x1875444569663488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573036.507962:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573036.507971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573036.507996:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569663488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573036.507997:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569663872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573036.508029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573036.508030:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573036.508037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788573036.508038:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788573036.508043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569663488:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573036.508044:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573036.508053:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9789 00000100:00080000:0.0:1788573036.508059:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573036.508064:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573036.508068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663616:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788573036.508072:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573036.508090:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569664000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573036.508097:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573036.508118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff920a48351500 x1875444569663616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573036.508129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569663616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573036.508135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573036.508138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573036.508140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663616:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573036.508144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573036.508157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573036.508164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff920a48353800 x1875444569663744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573036.508172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569663744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573036.508182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573036.508184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573036.508186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573036.508189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663872:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573036.508200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573036.508205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff920a48353b80 x1875444569663872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573036.508212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569663872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573036.508217:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573036.508219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573036.508221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569663872:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573036.508230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573036.508240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573036.508246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573036/real 1788573036] req@ffff920a48350a80 x1875444569664000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573036.508254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569664000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573036.508262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573036.508266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573036.508270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664000:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573036.955782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573037.531727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573037.979747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573038.555692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573039.003790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573039.579797:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573040.027734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573040.603704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573041.051794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573041.627750:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573041.627759:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573041.627763:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573041.627773:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9795 00000100:00100000:3.0:1788573041.627777:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573041.627782:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573041.627795:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573041.627833:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569664128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573041.627861:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573041.627864:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573041.627865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573041.627868:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9795 00000100:00080000:1.0:1788573041.627871:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573041.627874:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573041.627884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573041.627899:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569664256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573041.627912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573041.627918:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788573041.627919:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569664128:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573041.627922:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573041.627931:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9795 00000100:00080000:1.0:1788573041.627934:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573041.627938:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573041.627943:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573041.627954:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569664384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573041.627961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573041.627964:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573041.627965:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573041.627968:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9795 00000100:00080000:1.0:1788573041.627971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573041.627973:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573041.627976:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573041.627981:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573041.627987:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569664512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573041.627995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573041.627998:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573041.627999:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573041.628002:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9795 00000100:00080000:1.0:1788573041.628004:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573041.628007:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573041.628011:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573041.628019:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573041/real 1788573041] req@ffff920a52890000 x1875444569664128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573041.628020:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569664640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573041.628065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569664128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573041.628077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573041.628083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573041.628088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569664128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573041.628112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664256:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573041.628134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573041.628144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573041/real 1788573041] req@ffff920a52893b80 x1875444569664256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573041.628153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569664256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573041.628159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573041.628162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573041.628164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664256:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573041.628168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573041.628179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573041.628184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573041/real 1788573041] req@ffff920a52890380 x1875444569664384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573041.628192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569664384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573041.628197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573041.628199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573041.628201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573041.628205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664512:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573041.628216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573041.628221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573041/real 1788573041] req@ffff920a52891c00 x1875444569664512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573041.628229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569664512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573041.628234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573041.628235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573041.628237:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664512:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573041.628241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573041.628251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573041.628257:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573041/real 1788573041] req@ffff920a52893800 x1875444569664640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573041.628269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569664640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573041.628277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573041.628280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573041.628283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664640:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573042.075777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573042.651698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573043.099785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573043.675732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573044.123734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573044.699761:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573045.147775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573045.723681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573046.171731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573046.747712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573046.747713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573046.747722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573046.747726:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573046.747734:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9800 00000100:00080000:3.0:1788573046.747742:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573046.747749:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573046.747782:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569664768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573046.747810:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573046.747813:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573046.747815:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573046.747818:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9800 00000100:00080000:3.0:1788573046.747820:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573046.747823:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573046.747831:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573046.747839:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569664896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573046.747846:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573046.747848:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573046.747850:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573046.747853:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9800 00000100:00080000:3.0:1788573046.747855:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573046.747858:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573046.747861:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573046.747868:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569665024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573046.747875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573046.747877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573046.747880:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573046.747881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569664768:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573046.747885:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9800 00000100:00080000:3.0:1788573046.747890:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573046.747893:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573046.747897:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573046.747906:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569665152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573046.747913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573046.747915:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573046.747916:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573046.747919:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9800 00000100:00080000:3.0:1788573046.747934:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573046.747937:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573046.747939:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573046.747945:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573046.747949:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569665280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573046.747987:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573046/real 1788573046] req@ffff920a529b2a00 x1875444569664768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573046.748007:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569664768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573046.748014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573046.748017:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573046.748020:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569664768:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573046.748025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664896:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573046.748041:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573046.748049:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573046/real 1788573046] req@ffff920a529b2300 x1875444569664896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573046.748057:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569664896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573046.748062:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573046.748063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573046.748065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569664896:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573046.748090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573046.748102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573046.748109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573046/real 1788573046] req@ffff920a529b1c00 x1875444569665024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573046.748117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569665024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573046.748126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573046.748128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573046.748130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573046.748134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665152:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573046.748144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573046.748149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573046/real 1788573046] req@ffff920a529b0700 x1875444569665152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573046.748156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569665152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573046.748161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573046.748163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573046.748164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665152:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573046.748168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573046.748177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573046.748182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573046/real 1788573046] req@ffff920a529b3480 x1875444569665280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573046.748188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569665280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573071 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573046.748193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573046.748195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573046.748197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665280:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573047.195769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573047.771678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573048.219766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573048.795688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573049.243793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573049.819700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573050.267775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573050.843729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573051.291730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573051.867720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573051.867763:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573051.867772:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573051.867781:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573051.867793:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9805 00000100:00080000:3.0:1788573051.867805:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573051.867813:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573051.867859:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569665408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573051.867888:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573051.867893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573051.867896:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573051.867903:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9805 00000100:00080000:3.0:1788573051.867910:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573051.867917:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573051.867930:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573051.867931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569665408:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573051.867959:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569665536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573051.867974:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573051.867978:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573051.867980:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573051.867985:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9805 00000800:00000100:2.0:1788573051.868001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573051.868031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573051/real 1788573051] req@ffff920a529b2d80 x1875444569665408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573051.868048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569665408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573051.868058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573051.868062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788573051.868065:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573051.868066:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569665408:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573051.868074:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573051.868081:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573051.868095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665536:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573051.868096:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569665664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573051.868118:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573051.868122:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573051.868124:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573051.868128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573051.868146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573051/real 1788573051] req@ffff920a529b3100 x1875444569665536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573051.868155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569665536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573051.868161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573051.868163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573051.868166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665536:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573051.868174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573051.868187:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573051.868198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573051/real 1788573051] req@ffff920a529b0000 x1875444569665664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573051.868206:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569665664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573051.868211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573051.868213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573051.868215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573051.868270:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9805 00000100:00080000:3.0:1788573051.868276:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573051.868283:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573051.868336:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573051.868358:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569665792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573051.868385:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573051.868390:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573051.868393:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573051.868400:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9805 00000100:00100000:2.0:1788573051.868404:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665792:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788573051.868411:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573051.868420:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573051.868428:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573051.868433:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573051.868455:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573051/real 1788573051] req@ffff920a529b1f80 x1875444569665792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573051.868457:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569665920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573051.868497:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569665792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573051.868510:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573051.868515:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573051.868537:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573051.868553:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573051.868575:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573051.868626:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573051/real 1788573051] req@ffff920a529b0e00 x1875444569665920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573051.868643:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569665920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573076 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573051.868650:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573051.868653:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573051.868656:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569665920:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573052.315807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573052.891698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573053.339713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573053.915751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573054.363738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573054.939702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573055.387754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573055.963701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573056.411758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573056.987685:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573056.987686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573056.987695:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573056.987699:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573056.987708:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9810 00000100:00080000:3.0:1788573056.987717:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573056.987724:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573056.987758:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569666048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573056.987777:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573056.987781:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573056.987783:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573056.987787:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9810 00000100:00080000:3.0:1788573056.987792:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573056.987798:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573056.987808:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573056.987809:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569666048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573056.987830:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569666176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573056.987841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573056.987844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573056.987845:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573056.987849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9810 00000100:00080000:3.0:1788573056.987861:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573056.987864:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788573056.987869:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573056.987869:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573056.987884:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569666304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573056.987891:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573056.987895:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573056.987897:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573056.987902:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573056/real 1788573056] req@ffff920a529b3b80 x1875444569666048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573056.987906:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9810 00000100:00080000:3.0:1788573056.987916:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573056.987923:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573056.987925:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569666048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573056.987932:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573056.987936:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573056.987941:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573056.987944:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569666048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573056.987949:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569666432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573056.987957:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573056.987960:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573056.987961:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573056.987965:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666176:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573056.987968:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9810 00000100:00080000:3.0:1788573056.987986:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573056.987990:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573056.987995:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573056.988000:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573056.988006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573056/real 1788573056] req@ffff920a529b1500 x1875444569666176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573056.988013:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569666560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573056.988051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569666176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573056.988057:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573056.988060:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573056.988062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573056.988069:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573056.988081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573056.988087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573056/real 1788573056] req@ffff920a529b1180 x1875444569666304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573056.988104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569666304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573056.988109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573056.988112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573056.988114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573056.988117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666432:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573056.988128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573056.988134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573056/real 1788573056] req@ffff920a529b0380 x1875444569666432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573056.988145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569666432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573056.988159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573056.988163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573056.988166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573056.988184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573056.988204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573056.988213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573056/real 1788573056] req@ffff920a529b3b80 x1875444569666560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573056.988220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569666560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573081 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573056.988226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573056.988228:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573056.988230:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666560:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573057.435771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573058.011705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573058.459729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573059.035687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573059.483767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573060.059812:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573060.507726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573061.083720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573061.531784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573062.107694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573062.107708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.107715:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573062.107719:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.107728:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9815 00000100:00080000:0.0:1788573062.107737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.107744:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573062.107778:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569666688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.107800:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.107803:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573062.107805:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.107808:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9815 00000100:00080000:0.0:1788573062.107810:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.107812:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573062.107820:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573062.107833:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569666816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.107841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.107845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573062.107846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.107849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9815 00000100:00080000:0.0:1788573062.107851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.107859:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573062.107861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569666688:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573062.107863:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573062.107885:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569666944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.107892:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.107895:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573062.107897:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.107901:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9815 00000100:00080000:0.0:1788573062.107903:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.107915:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573062.107918:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573062.107930:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569667072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573062.107936:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573062.107938:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573062.107940:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573062.107942:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9815 00000100:00080000:0.0:1788573062.107946:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573062.107948:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573062.107951:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573062.107961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573062.107963:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569667200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573062.108018:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573062] req@ffff920a48353480 x1875444569666688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573062.108037:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569666688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573062.108046:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573062.108051:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573062.108054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569666688:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573062.108079:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666816:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573062.108099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573062.108108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573062] req@ffff920a48350380 x1875444569666816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573062.108116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569666816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573062.108122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573062.108125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573062.108127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573062.108132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573062.108144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573062.108150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573062] req@ffff920a48350700 x1875444569666944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573062.108157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569666944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573062.108163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573062.108165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573062.108167:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569666944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573062.108170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667072:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573062.108183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573062.108189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573062] req@ffff920a48350a80 x1875444569667072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573062.108197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569667072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573062.108202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573062.108204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573062.108206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573062.108209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573062.108220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573062.108225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573062/real 1788573062] req@ffff920a48353b80 x1875444569667200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573062.108233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569667200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573062.108238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573062.108240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573062.108242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667200:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573062.555754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573063.131666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573063.579718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573064.155706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573064.603724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573065.179665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573065.627730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573066.203672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573066.651740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573067.227700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573067.227725:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573067.227732:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573067.227735:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573067.227744:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9820 00000100:00080000:0.0:1788573067.227753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573067.227760:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573067.227795:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569667328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573067.227819:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573067.227822:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573067.227823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573067.227826:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9820 00000100:00080000:0.0:1788573067.227829:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573067.227831:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573067.227839:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573067.227847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569667456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573067.227854:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573067.227857:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573067.227859:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573067.227861:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9820 00000100:00080000:0.0:1788573067.227863:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573067.227865:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573067.227868:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573067.227875:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569667584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573067.227883:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569667328:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573067.227885:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573067.227892:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573067.227895:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573067.227900:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9820 00000100:00080000:0.0:1788573067.227902:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573067.227904:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573067.227907:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573067.227917:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569667712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573067.227923:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573067.227925:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573067.227926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573067.227928:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9820 00000100:00080000:0.0:1788573067.227931:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573067.227933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573067.227936:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573067.227942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569667840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573067.227956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573067.227995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff920a48353800 x1875444569667328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573067.228013:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569667328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573067.228022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573067.228026:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573067.228030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569667328:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573067.228036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667456:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573067.228059:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573067.228071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff920a48351500 x1875444569667456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573067.228083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569667456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573067.228091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573067.228095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573067.228098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573067.228121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573067.228146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573067.228155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff920a48351c00 x1875444569667584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573067.228163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569667584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573067.228173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573067.228175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573067.228177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573067.228181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667712:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573067.228192:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573067.228198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff920a48353b80 x1875444569667712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573067.228205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569667712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573067.228210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573067.228212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573067.228214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573067.228217:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573067.228227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573067.228232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573067/real 1788573067] req@ffff920a48350a80 x1875444569667840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573067.228265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569667840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573067.228271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573067.228273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573067.228276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569667840:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573067.675747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573068.251682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573068.699719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573069.275700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573069.723790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573070.299668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573070.747696:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573071.323658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573071.771766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573072.347660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573072.347754:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.347759:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573072.347763:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.347770:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9825 00000100:00080000:3.0:1788573072.347780:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.347786:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573072.347817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569667968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.347838:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.347842:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573072.347844:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.347848:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9825 00000100:00080000:3.0:1788573072.347852:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.347857:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573072.347867:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573072.347880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569667968:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573072.347880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569668096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.347890:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.347894:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573072.347896:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.347899:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9825 00000100:00080000:3.0:1788573072.347902:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.347906:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573072.347910:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573072.347919:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569668224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.347927:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.347931:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573072.347932:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573072.347936:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9825 00000800:00000100:2.0:1788573072.347938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573072.347939:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.347943:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573072.347947:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573072.347955:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569668352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573072.347963:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573072.347966:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573072.347968:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573072.347970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff920a529b0e00 x1875444569667968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573072.347971:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9825 00000100:00080000:3.0:1788573072.347975:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573072.347978:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573072.347989:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569667968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573072.347989:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573072.347999:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573072.347999:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569668480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573072.348004:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573072.348008:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569667968:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573072.348030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668096:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573072.348053:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573072.348064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff920a529b1f80 x1875444569668096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573072.348073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569668096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573072.348079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573072.348081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573072.348083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668096:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573072.348087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573072.348119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573072.348127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff920a529b0000 x1875444569668224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573072.348135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569668224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573072.348141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573072.348144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573072.348146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573072.348149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668352:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573072.348160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573072.348165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff920a529b3100 x1875444569668352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573072.348173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569668352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573072.348178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573072.348179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573072.348181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573072.348184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573072.348193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573072.348198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573072/real 1788573072] req@ffff920a529b2d80 x1875444569668480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573072.348210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569668480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573072.348215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573072.348216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573072.348218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573072.795778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573073.371924:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573073.819815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573074.395685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573074.843784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573075.419723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573075.867794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573076.443720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573076.891713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573077.467689:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573077.467697:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573077.467701:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573077.467711:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9830 00000100:00080000:3.0:1788573077.467722:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573077.467728:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573077.467735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788573077.467764:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569668608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573077.467785:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573077.467788:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573077.467790:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573077.467793:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9830 00000100:00080000:3.0:1788573077.467795:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573077.467797:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573077.467805:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573077.467813:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569668736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573077.467821:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573077.467824:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569668608:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573077.467825:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573077.467828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573077.467834:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9830 00000100:00080000:3.0:1788573077.467836:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573077.467838:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573077.467842:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573077.467851:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569668864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573077.467858:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573077.467861:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573077.467863:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573077.467866:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9830 00000100:00080000:3.0:1788573077.467877:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573077.467879:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573077.467883:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573077.467891:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569668992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573077.467910:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573077.467913:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573077.467915:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573077.467919:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9830 00000100:00080000:3.0:1788573077.467922:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573077.467925:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573077.467929:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573077.467932:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668736:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573077.467939:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569669120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573077.467968:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573077.467989:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668992:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573077.468009:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669120:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573077.915781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573078.491763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573078.939786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573079.515798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573079.963832:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573080.539864:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573080.539923:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573080.540014:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573080.540037:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573080.540104:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573080.540132:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573080.540142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573080] req@ffff920a529b0700 x1875444569668736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788573080.540151:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573080.540170:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573080.540196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569668736/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573080.540213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573080.540219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573080.540224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668736:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573080.540234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573080] req@ffff920a529b1c00 x1875444569668864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573080.540251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569668864/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573080.540260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573080.540264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573080.540269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573080.540276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573080] req@ffff920a529b2300 x1875444569668992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573080.540289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569668992/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573080.540297:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573080.540301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573080.540305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569668992:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573080.540312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573080] req@ffff920a529b3480 x1875444569668608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573080.540326:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569668608/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573080.540333:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573080.540336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573080.540339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569668608:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573080.540370:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573077/real 1788573080] req@ffff920a529b2a00 x1875444569669120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573080.540382:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569669120/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573080.540390:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573080.540393:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573080.540396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669120:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573080.987740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573081.563759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573082.011790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573082.587734:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.587746:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573082.587752:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.587768:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9835 00000100:00080000:0.0:1788573082.587782:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573082.587789:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573082.587793:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573082.587838:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569669248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.587869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.587873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573082.587875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.587881:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9835 00000100:00080000:0.0:1788573082.587886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573082.587892:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573082.587906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573082.587909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569669248:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573082.587924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569669376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.587937:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.587942:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573082.587944:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.587948:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9835 00000100:00080000:0.0:1788573082.587952:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573082.587957:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573082.587964:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573082.587978:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569669504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573082.587996:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788573082.588001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573082.588002:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573082.588007:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.588013:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9835 00000100:00080000:0.0:1788573082.588017:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573082.588023:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573082.588030:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573082.588046:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569669632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573082.588055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff920a48350700 x1875444569669248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573082.588061:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573082.588070:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573082.588075:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573082.588081:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9835 00000100:00080000:0.0:1788573082.588087:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573082.588091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569669248/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573082.588096:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573082.588104:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573082.588113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573082.588121:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569669760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573082.588123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573082.588132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569669248:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573082.588171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669376:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573082.588199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573082.588213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff920a48350380 x1875444569669376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573082.588225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569669376/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573082.588234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573082.588237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573082.588240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669376:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573082.588246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573082.588267:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573082.588277:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff920a48353480 x1875444569669504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573082.588288:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569669504/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573082.588296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573082.588299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573082.588302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573082.588308:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669632:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573082.588328:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573082.588340:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff920a48350a80 x1875444569669632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573082.588352:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569669632/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573082.588361:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573082.588364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573082.588368:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669632:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573082.588374:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573082.588394:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573082.588405:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573082/real 1788573082] req@ffff920a48353b80 x1875444569669760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573082.588416:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569669760/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573082.588425:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573082.588428:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573082.588432:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569669760:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573083.035699:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573083.611802:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573084.059767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573084.635668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573085.083822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573085.659709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573086.107750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573086.683676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573087.131755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573087.707666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573087.707738:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573087.707750:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573087.707756:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573087.707770:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9841 00000100:00080000:2.0:1788573087.707784:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573087.707795:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573087.707851:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569669888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573087.707893:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573087.707897:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573087.707899:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573087.707905:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9841 00000100:00080000:2.0:1788573087.707915:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573087.707920:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573087.707932:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573087.707940:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569669888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573087.707950:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569670016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573087.707961:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573087.707964:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573087.707966:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573087.707969:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9841 00000100:00080000:2.0:1788573087.707972:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573087.707976:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573087.707979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573087.707990:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569670144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573087.707996:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573087.707999:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573087.708000:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573087.708003:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9841 00000100:00080000:2.0:1788573087.708005:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573087.708008:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573087.708011:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573087.708019:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573087.708021:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569670272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573087.708030:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573087.708034:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573087.708036:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573087.708039:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9841 00000100:00080000:2.0:1788573087.708051:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573087.708054:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573087.708057:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573087.708065:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569670400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573087.708069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573087/real 1788573087] req@ffff920a60ac4e00 x1875444569669888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.708104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569669888/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.708120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.708124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573087.708127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569669888:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573087.708152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670016:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573087.708171:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573087.708183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573087/real 1788573087] req@ffff920a60ac5c00 x1875444569670016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.708193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569670016/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.708201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.708205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573087.708209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670016:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573087.708212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573087.708228:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573087.708237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573087/real 1788573087] req@ffff920a60ac4a80 x1875444569670144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.708247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569670144/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.708258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.708261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573087.708263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573087.708268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670272:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573087.708284:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573087.708293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573087/real 1788573087] req@ffff920a60ac4000 x1875444569670272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.708301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569670272/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.708309:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.708313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573087.708317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670272:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573087.708326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573087.708345:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573087.708362:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573087/real 1788573087] req@ffff920a60ac4380 x1875444569670400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573087.708370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569670400/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573112 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573087.708375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573087.708377:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573087.708380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670400:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573088.155784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573088.731721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573089.179771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573089.755670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573090.203749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573090.779721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573091.227791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573091.803661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573092.251737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573092.827690:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573092.827693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573092.827699:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573092.827704:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573092.827712:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9846 00000100:00080000:2.0:1788573092.827721:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573092.827727:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573092.827759:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569670528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573092.827781:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573092.827784:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573092.827785:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573092.827788:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9846 00000100:00080000:2.0:1788573092.827791:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573092.827793:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573092.827800:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573092.827808:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569670656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573092.827815:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573092.827818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573092.827819:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573092.827823:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9846 00000100:00080000:2.0:1788573092.827826:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573092.827828:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573092.827832:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573092.827841:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569670784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573092.827842:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569670528:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573092.827852:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573092.827855:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573092.827856:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573092.827859:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9846 00000100:00080000:2.0:1788573092.827861:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573092.827863:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573092.827866:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573092.827875:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569670912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573092.827881:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573092.827883:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573092.827884:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573092.827887:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9846 00000100:00080000:2.0:1788573092.827889:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573092.827890:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573092.827893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573092.827899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569671040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573092.827928:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573092.827964:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573092/real 1788573092] req@ffff920a60ac4700 x1875444569670528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573092.827984:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569670528/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573092.827996:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573092.828000:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573092.828004:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569670528:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573092.828012:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670656:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573092.828036:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573092.828050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573092/real 1788573092] req@ffff920a60ac7800 x1875444569670656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573092.828063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569670656/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573092.828072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573092.828076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573092.828080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670656:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573092.828106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573092.828127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573092.828139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573092/real 1788573092] req@ffff920a60ac5500 x1875444569670784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573092.828151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569670784/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573092.828161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573092.828164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573092.828168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573092.828174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670912:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573092.828193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573092.828204:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573092/real 1788573092] req@ffff920a60ac6300 x1875444569670912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573092.828217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569670912/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573092.828225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573092.828228:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573092.828232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569670912:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573092.828238:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573092.828257:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573092.828278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573092/real 1788573092] req@ffff920a60ac7100 x1875444569671040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573092.828291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569671040/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573117 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573092.828300:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573092.828303:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573092.828323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671040:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573093.275754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573093.851684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573094.299694:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573094.875710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573095.323739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573095.899686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573096.347748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573096.923681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573097.371729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573097.947684:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573097.947696:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573097.947701:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573097.947704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573097.947714:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9851 00000100:00080000:1.0:1788573097.947726:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573097.947734:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573097.947773:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569671168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573097.947805:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573097.947809:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573097.947811:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573097.947816:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9851 00000100:00080000:1.0:1788573097.947820:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573097.947824:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573097.947834:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573097.947850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569671296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573097.947861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569671168:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573097.947865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573097.947871:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573097.947873:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573097.947879:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9851 00000100:00080000:1.0:1788573097.947883:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573097.947887:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573097.947893:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573097.947907:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569671424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573097.947918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573097.947922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573097.947924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573097.947930:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9851 00000800:00000100:0.0:1788573097.947932:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573097.947935:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573097.947940:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573097.947946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573097.947958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569671552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573097.947969:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573097.947975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573097/real 1788573097] req@ffff920a52890a80 x1875444569671168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573097.947975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573097.947979:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573097.947987:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9851 00000100:00080000:1.0:1788573097.947993:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573097.947999:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573097.948005:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573097.948006:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569671168/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573097.948021:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569671680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573097.948022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573097.948032:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573097.948041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569671168:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573097.948074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671296:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573097.948094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573097.948103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573097/real 1788573097] req@ffff920a52892680 x1875444569671296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573097.948125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569671296/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573097.948132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573097.948135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573097.948137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671296:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573097.948142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573097.948157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573097.948164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573097/real 1788573097] req@ffff920a52891500 x1875444569671424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573097.948172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569671424/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573097.948178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573097.948180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573097.948182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573097.948185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671552:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573097.948196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573097.948201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573097/real 1788573097] req@ffff920a52891f80 x1875444569671552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573097.948208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569671552/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573097.948212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573097.948214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573097.948217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671552:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573097.948220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573097.948229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573097.948234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573097/real 1788573097] req@ffff920a52892a00 x1875444569671680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573097.948241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569671680/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573122 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573097.948255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573097.948257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573097.948259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671680:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573098.395747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573098.971707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573099.419757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573099.995674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573100.443809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573101.019664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573101.467739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573102.043669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573102.491803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573103.067674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573103.067693:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.067699:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573103.067703:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.067711:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9856 00000100:00080000:2.0:1788573103.067719:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.067726:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573103.067759:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569671808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.067783:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.067786:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573103.067787:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.067791:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9856 00000100:00080000:2.0:1788573103.067793:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.067796:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573103.067804:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573103.067813:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569671936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.067821:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.067825:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573103.067828:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.067832:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9856 00000100:00080000:2.0:1788573103.067834:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.067837:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573103.067840:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573103.067849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569672064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573103.067861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569671808:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573103.067863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.067871:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573103.067875:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.067881:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9856 00000100:00080000:2.0:1788573103.067883:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.067885:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573103.067888:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573103.067897:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569672192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573103.067903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573103.067906:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573103.067907:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573103.067910:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9856 00000100:00080000:2.0:1788573103.067912:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573103.067929:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573103.067935:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573103.067949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569672320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573103.067996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573103.068050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff920a60ac6a00 x1875444569671808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.068069:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569671808/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573103.068080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573103.068084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573103.068087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569671808:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573103.068096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671936:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573103.068124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573103.068139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff920a60ac5180 x1875444569671936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.068153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569671936/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573103.068161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573103.068163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573103.068166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569671936:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573103.068193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573103.068209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573103.068217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff920a60ac7100 x1875444569672064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.068225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569672064/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573103.068231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573103.068233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573103.068235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573103.068239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672192:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573103.068251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573103.068256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff920a60ac6300 x1875444569672192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.068264:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569672192/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573103.068270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573103.068272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573103.068274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672192:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573103.068277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573103.068287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573103.068293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573103/real 1788573103] req@ffff920a60ac5500 x1875444569672320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573103.068301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569672320/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573103.068306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573103.068308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573103.068310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672320:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573103.515764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573104.091696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573104.539793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573105.115737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573105.563849:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573106.139709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573106.587752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573107.163686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573107.611738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573108.187688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573108.187739:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573108.187751:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573108.187757:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573108.187769:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9861 00000100:00080000:1.0:1788573108.187788:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573108.187799:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573108.187846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569672448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573108.187878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573108.187882:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573108.187885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573108.187890:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9861 00000100:00080000:1.0:1788573108.187894:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573108.187900:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573108.187911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573108.187927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569672576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573108.187934:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569672448:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573108.187957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573108.187960:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573108.187961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573108.187967:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9861 00000100:00080000:1.0:1788573108.187973:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573108.187976:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573108.187981:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573108.187990:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569672704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573108.187997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573108.188000:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573108.188002:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573108.188005:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9861 00000100:00080000:1.0:1788573108.188008:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573108.188010:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573108.188014:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573108.188015:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573108.188028:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569672832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573108.188035:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573108.188038:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573108.188041:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573108.188046:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9861 00000100:00080000:1.0:1788573108.188051:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573108.188054:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573108.188059:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573108.188060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff920a52893480 x1875444569672448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573108.188070:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569672960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573108.188088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569672448/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573108.188103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573108.188107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573108.188111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569672448:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573108.188134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672576:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573108.188151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573108.188160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff920a52890e00 x1875444569672576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573108.188169:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569672576/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573108.188174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573108.188177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573108.188181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672576:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573108.188187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573108.188208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573108.188220:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff920a52891180 x1875444569672704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573108.188234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569672704/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573108.188241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573108.188244:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573108.188246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573108.188251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672832:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573108.188268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573108.188276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff920a52890700 x1875444569672832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573108.188284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569672832/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573108.188289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573108.188291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573108.188293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672832:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573108.188296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573108.188314:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573108.188324:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573108/real 1788573108] req@ffff920a52893100 x1875444569672960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573108.188337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569672960/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573108.188346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573108.188350:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573108.188354:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569672960:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573108.635817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573109.211726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573109.659780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573110.235725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573110.683802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573111.259704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573111.707781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573112.283712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573112.731789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573113.307683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573113.307724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.307732:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573113.307736:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.307746:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9866 00000100:00080000:1.0:1788573113.307756:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.307764:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573113.307803:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569673088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.307832:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.307836:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573113.307839:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.307844:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9866 00000100:00080000:1.0:1788573113.307849:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.307857:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573113.307858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569673088:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573113.307868:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573113.307886:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569673216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.307898:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.307902:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573113.307904:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.307909:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9866 00000100:00080000:1.0:1788573113.307914:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.307919:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573113.307924:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573113.307928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573113.307946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569673344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.307958:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573113.307961:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff920a52891880 x1875444569673088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573113.307962:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573113.307967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573113.307977:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9866 00000100:00080000:1.0:1788573113.307983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.307988:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573113.307991:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569673088/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573113.307997:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573113.308002:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573113.308007:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573113.308011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569673088:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573113.308016:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569673472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573113.308029:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573113.308032:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573113.308035:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573113.308038:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673216:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573113.308044:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9866 00000100:00080000:1.0:1788573113.308050:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573113.308055:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573113.308062:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573113.308066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573113.308077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff920a52892d80 x1875444569673216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573113.308087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569673216/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573113.308093:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569673600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573113.308096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573113.308104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573113.308110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673216:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573113.308120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573113.308136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573113.308144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff920a52892300 x1875444569673344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573113.308165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569673344/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573113.308174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573113.308177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573113.308181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573113.308188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673472:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573113.308207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573113.308215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff920a52893100 x1875444569673472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573113.308223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569673472/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573113.308229:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573113.308231:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573113.308233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673472:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573113.308246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573113.308265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573113.308276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573113/real 1788573113] req@ffff920a52890700 x1875444569673600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573113.308289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569673600/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573113.308302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573113.308304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573113.308306:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673600:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573113.755761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573114.331649:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573114.779781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573115.355679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573115.803754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573116.379705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573116.827795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573117.403705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573117.851705:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573118.427701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573118.427724:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.427732:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573118.427736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.427746:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9871 00000100:00080000:2.0:1788573118.427756:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.427765:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573118.427803:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569673728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.427832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573118.427836:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573118.427838:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.427842:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9871 00000100:00080000:2.0:1788573118.427846:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.427851:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573118.427861:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573118.427874:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569673856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.427896:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573118.427897:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569673728:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573118.427905:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573118.427909:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.427914:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9871 00000100:00080000:2.0:1788573118.427919:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.427923:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573118.427930:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573118.427946:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569673984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573118.427956:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788573118.427957:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573118.427964:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573118.427966:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.427972:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9871 00000100:00080000:2.0:1788573118.427977:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.427982:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573118.427989:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573118.427990:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff920a60ac7800 x1875444569673728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573118.428006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569674112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573118.428009:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569673728/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573118.428037:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573118.428037:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573118.428043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788573118.428045:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788573118.428048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569673728:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573118.428050:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573118.428059:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9871 00000100:00080000:2.0:1788573118.428064:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573118.428069:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573118.428074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673856:192.168.202.157@tcp:38:kworker.0 00000008:00100000:2.0:1788573118.428077:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573118.428094:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569674240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573118.428107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573118.428124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff920a60ac4700 x1875444569673856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573118.428139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569673856/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573118.428148:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573118.428152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573118.428155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673856:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573118.428161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573118.428180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573118.428191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff920a60ac4380 x1875444569673984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573118.428202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569673984/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573118.428210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573118.428213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573118.428217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569673984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573118.428222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674112:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573118.428239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573118.428248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff920a60ac4000 x1875444569674112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573118.428259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569674112/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573118.428266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573118.428270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573118.428273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674112:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573118.428289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573118.428310:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573118.428321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573118/real 1788573118] req@ffff920a60ac4a80 x1875444569674240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573118.428365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569674240/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573118.428371:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573118.428373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573118.428375:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674240:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573118.875747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573119.451666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573119.899738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573120.475690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573120.923781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573121.499661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573121.947801:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573122.523690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573122.971726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573123.547682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573123.547713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573123.547723:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573123.547727:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573123.547737:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9876 00000100:00080000:0.0:1788573123.547747:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573123.547753:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573123.547786:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569674368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573123.547808:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573123.547812:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573123.547814:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573123.547818:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9876 00000100:00080000:0.0:1788573123.547821:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573123.547827:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573123.547831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569674368:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573123.547835:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573123.547847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569674496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573123.547857:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573123.547860:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573123.547862:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573123.547864:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9876 00000100:00080000:0.0:1788573123.547867:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573123.547877:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573123.547882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573123.547893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569674624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573123.547900:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573123.547904:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788573123.547907:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573123.547908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573123.547916:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9876 00000100:00080000:0.0:1788573123.547921:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573123.547928:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573123.547932:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573123.547944:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569674752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573123.547952:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573123.547954:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff920a48351c00 x1875444569674368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573123.547958:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573123.547963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573123.547967:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9876 00000100:00080000:0.0:1788573123.547970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573123.547978:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573123.547988:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573123.547998:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569674368/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573123.548004:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569674880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573123.548023:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573123.548034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573123.548040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569674368:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573123.548076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674496:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573123.548105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573123.548120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff920a48351500 x1875444569674496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573123.548149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569674496/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573123.548158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573123.548162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573123.548166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674496:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573123.548173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573123.548204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573123.548217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff920a48353800 x1875444569674624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573123.548230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569674624/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573123.548247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573123.548252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573123.548256:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573123.548263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674752:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573123.548287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573123.548300:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff920a48353b80 x1875444569674752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573123.548319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569674752/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573123.548328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573123.548332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573123.548336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674752:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573123.548344:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573123.548372:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573123.548391:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573123/real 1788573123] req@ffff920a48350a80 x1875444569674880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573123.548405:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569674880/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573123.548414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573123.548418:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573123.548424:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569674880:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573123.995779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573124.571748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573125.019752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573125.595721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573126.043784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573126.619695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573127.067721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573127.643706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573128.091810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573128.667671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573128.667727:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573128.667738:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573128.667742:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573128.667752:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9882 00000100:00080000:0.0:1788573128.667764:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573128.667772:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573128.667813:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569675008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573128.667844:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573128.667848:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573128.667850:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573128.667855:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9882 00000100:00080000:0.0:1788573128.667859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573128.667864:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573128.667876:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573128.667892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569675136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573128.667906:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573128.667911:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573128.667913:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573128.667923:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9882 00000100:00100000:1.0:1788573128.667923:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569675008:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573128.667965:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573128.667974:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573128.667981:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573128.667999:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569675264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573128.668010:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573128.668014:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573128.668016:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573128.668021:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9882 00000100:00080000:0.0:1788573128.668026:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573128.668031:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573128.668035:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573128.668045:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573128.668066:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569675392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573128.668077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573128/real 1788573128] req@ffff920a48353480 x1875444569675008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573128.668081:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573128.668098:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573128.668103:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573128.668113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569675008/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573128.668114:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9882 00000100:00080000:0.0:1788573128.668128:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573128.668138:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573128.668141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788573128.668147:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573128.668152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573128.668160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569675008:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573128.668168:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569675520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573128.668196:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675136:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573128.668222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573128.668232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573128/real 1788573128] req@ffff920a48350380 x1875444569675136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573128.668241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569675136/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573128.668247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573128.668250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573128.668252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675136:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573128.668257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573128.668268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573128.668274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573128/real 1788573128] req@ffff920a48350700 x1875444569675264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573128.668282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569675264/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573128.668287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573128.668289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573128.668291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573128.668294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675392:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573128.668306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573128.668312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573128/real 1788573128] req@ffff920a48350a80 x1875444569675392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573128.668319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569675392/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573128.668323:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573128.668325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573128.668327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675392:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573128.668330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573128.668340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573128.668346:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573128/real 1788573128] req@ffff920a48353b80 x1875444569675520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573128.668353:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569675520/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573128.668358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573128.668360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573128.668363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675520:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573129.116125:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573129.691697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573130.139790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573130.715741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573131.163824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573131.739732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573132.187740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573132.763696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573133.211749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573133.787699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573133.787700:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573133.787711:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573133.787716:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573133.787727:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9887 00000100:00080000:0.0:1788573133.787768:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573133.787777:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573133.787817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569675648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573133.787848:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573133.787853:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573133.787855:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573133.787859:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9887 00000100:00080000:0.0:1788573133.787863:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573133.787868:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573133.787879:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573133.787893:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569675648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573133.787899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569675776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573133.787913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573133.787917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573133.787920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573133.787925:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9887 00000100:00080000:0.0:1788573133.787941:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573133.787947:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573133.787952:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573133.787954:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573133.787972:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569675904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573133.787984:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573133.787985:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573133/real 1788573133] req@ffff920a48353800 x1875444569675648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573133.787995:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573133.788000:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573133.788007:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9887 00000100:00100000:1.0:1788573133.788007:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569675648/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573133.788016:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573133.788021:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573133.788026:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573133.788029:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:0.0:1788573133.788036:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573133.788037:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569675648:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573133.788057:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569676032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573133.788070:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675776:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573133.788072:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573133.788080:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573133.788083:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573133.788088:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9887 00000100:00080000:0.0:1788573133.788098:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788573133.788104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573133.788105:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573133.788113:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573133.788123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573133/real 1788573133] req@ffff920a48351500 x1875444569675776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573133.788131:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569676160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573133.788148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569675776/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573133.788160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573133.788164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573133.788169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675776:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573133.788177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573133.788203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573133.788222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573133/real 1788573133] req@ffff920a48351c00 x1875444569675904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573133.788240:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569675904/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573133.788250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573133.788255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573133.788259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569675904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573133.788272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676032:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573133.788288:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573133.788295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573133/real 1788573133] req@ffff920a48353b80 x1875444569676032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573133.788303:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569676032/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573133.788308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573133.788310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573133.788312:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676032:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573133.788315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573133.788327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573133.788337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573133/real 1788573133] req@ffff920a48350a80 x1875444569676160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573133.788350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569676160/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573158 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573133.788358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573133.788361:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573133.788364:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676160:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573134.235868:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573134.811722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573135.259766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573135.835698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573136.283785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573136.859685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573137.307762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573137.883703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573138.331780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573138.907699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573138.907713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573138.907719:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573138.907722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573138.907730:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9892 00000100:00080000:1.0:1788573138.907739:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573138.907745:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573138.907778:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569676288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573138.907810:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573138.907814:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573138.907817:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573138.907822:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9892 00000100:00080000:1.0:1788573138.907826:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573138.907831:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573138.907842:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573138.907852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569676288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573138.907854:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569676416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573138.907869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573138.907873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573138.907874:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573138.907879:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9892 00000100:00080000:1.0:1788573138.907882:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573138.907886:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573138.907890:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573138.907900:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569676544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573138.907908:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573138.907912:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573138.907913:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573138.907917:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9892 00000100:00080000:1.0:1788573138.907920:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573138.907924:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573138.907924:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573138.907928:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573138.907937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569676672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573138.907945:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573138.907949:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573138.907950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573138.907954:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9892 00000100:00100000:0.0:1788573138.907957:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573138/real 1788573138] req@ffff920a52891180 x1875444569676288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573138.907958:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573138.907961:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573138.907966:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573138.907975:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569676800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573138.907976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569676288/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573138.907987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573138.907991:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573138.907995:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569676288:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573138.908027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676416:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573138.908047:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573138.908056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573138/real 1788573138] req@ffff920a52890e00 x1875444569676416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573138.908065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569676416/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573138.908075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573138.908078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573138.908088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676416:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573138.908092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573138.908104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573138.908111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573138/real 1788573138] req@ffff920a52893480 x1875444569676544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573138.908119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569676544/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573138.908124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573138.908126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573138.908128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573138.908131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676672:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573138.908142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573138.908148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573138/real 1788573138] req@ffff920a52892a00 x1875444569676672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573138.908156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569676672/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573138.908161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573138.908162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573138.908165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676672:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573138.908168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573138.908177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573138.908182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573138/real 1788573138] req@ffff920a52891f80 x1875444569676800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573138.908189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569676800/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573163 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573138.908193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573138.908195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573138.908197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569676800:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573139.355789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573139.931718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573140.379797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573140.955684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573141.403754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573141.979687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573142.427756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573143.003709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573143.451805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573144.027730:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573144.027740:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1788573144.027741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573144.027745:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573144.027759:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9897 00000100:00080000:1.0:1788573144.027768:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573144.027774:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573144.027807:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569676928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573144.027836:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573144.027839:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573144.027840:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573144.027843:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9897 00000100:00080000:1.0:1788573144.027846:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573144.027848:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573144.027856:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573144.027864:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569677056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573144.027872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573144.027877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573144.027878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573144.027884:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9897 00000100:00080000:1.0:1788573144.027886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573144.027888:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573144.027891:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573144.027899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569677184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573144.027907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573144.027915:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569676928:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573144.027915:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573144.027920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573144.027932:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9897 00000100:00080000:1.0:1788573144.027936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573144.027939:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573144.027942:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573144.027955:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569677312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573144.027961:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573144.027964:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573144.027965:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573144.027969:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9897 00000100:00080000:1.0:1788573144.027971:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573144.027974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573144.027978:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573144.027985:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569677440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573144.028040:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677056:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573144.028093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573144.028124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677312:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573144.028140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677440:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573144.475753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573145.051691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573145.499803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573146.075731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573146.523783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573147.099701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573147.099922:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573147.099971:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573147.099988:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573147.100037:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573147.100050:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573147.100063:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573147.100082:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788573147.100099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573147] req@ffff920a52890a80 x1875444569677184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573147.100138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569677184/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573147.100150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573147.100156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573147.100160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573147.100169:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573147] req@ffff920a52893800 x1875444569677312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573147.100184:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569677312/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573147.100192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573147.100196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573147.100200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677312:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573147.100208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573147] req@ffff920a52891c00 x1875444569677440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573147.100219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569677440/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573147.100227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573147.100230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573147.100234:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573147.100244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573147] req@ffff920a52891500 x1875444569676928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573147.100260:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569676928/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573147.100266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573147.100299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573147.100302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569676928:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573147.100309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573144/real 1788573147] req@ffff920a52892680 x1875444569677056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573147.100322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569677056/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573147.100330:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573147.100334:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573147.100352:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677056:192.168.202.157@tcp:38:kworker.0 00010000:00010000:0.0:1788573147.547793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573148.123721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573148.571784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573149.147738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573149.147819:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573149.147831:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573149.147835:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573149.147859:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9902 00000100:00080000:2.0:1788573149.147871:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573149.147881:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573149.147927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569677568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573149.147961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573149.147965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573149.147967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573149.147972:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9902 00000100:00080000:2.0:1788573149.147976:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573149.147981:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573149.147994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573149.148010:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569677696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573149.148021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569677568:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573149.148026:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573149.148032:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573149.148035:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573149.148041:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9902 00000100:00080000:2.0:1788573149.148045:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573149.148050:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573149.148055:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573149.148072:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569677824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573149.148084:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573149.148088:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573149.148090:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573149.148094:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9902 00000100:00080000:2.0:1788573149.148101:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573149.148106:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573149.148112:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573149.148114:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573149.148130:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569677952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573149.148150:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573149.148154:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573149.148156:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573149.148161:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9902 00000100:00100000:0.0:1788573149.148167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff920a60ac5c00 x1875444569677568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573149.148168:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573149.148178:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573149.148190:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573149.148216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569677568/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.148232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573149.148241:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569678080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573149.148243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573149.148253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569677568:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573149.148291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677696:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573149.148320:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573149.148334:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff920a60ac4e00 x1875444569677696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.148346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569677696/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.148355:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.148358:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573149.148361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677696:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573149.148368:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573149.148390:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573149.148400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff920a60ac4a80 x1875444569677824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.148411:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569677824/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.148419:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.148423:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573149.148427:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573149.148433:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677952:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573149.148459:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573149.148485:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff920a60ac4000 x1875444569677952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.148497:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569677952/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.148505:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.148507:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573149.148510:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569677952:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573149.148516:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573149.148536:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573149.148547:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573149/real 1788573149] req@ffff920a60ac4380 x1875444569678080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573149.148558:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569678080/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573149.148566:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573149.148569:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573149.148572:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678080:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573149.595798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573150.171683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573150.619777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573151.195701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573151.643838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573152.219722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573152.667781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573153.243702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573153.691806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573154.267770:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573154.267773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573154.267782:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573154.267788:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573154.267800:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9907 00000100:00080000:1.0:1788573154.267812:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573154.267820:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573154.267860:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569678208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573154.267889:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573154.267893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573154.267895:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573154.267900:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9907 00000100:00080000:1.0:1788573154.267903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573154.267909:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573154.267920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573154.267934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569678336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573154.267948:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569678208:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573154.267952:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573154.267965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573154.267967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573154.267973:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9907 00000100:00080000:1.0:1788573154.267988:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573154.267992:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573154.267998:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573154.268014:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569678464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573154.268038:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573154.268043:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573154.268045:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573154.268050:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9907 00000100:00080000:1.0:1788573154.268093:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573154.268099:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573154.268105:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573154.268118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573154.268126:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569678592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573154.268140:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573154.268144:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573154.268146:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573154.268151:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9907 00000100:00080000:1.0:1788573154.268156:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573154.268162:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573154.268171:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573154.268182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff920a52890380 x1875444569678208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573154.268192:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569678720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573154.268214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569678208/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.268232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.268238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573154.268263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569678208:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573154.268298:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678336:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573154.268327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573154.268340:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff920a52893b80 x1875444569678336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.268352:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569678336/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.268361:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.268364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573154.268368:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678336:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573154.268373:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573154.268393:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573154.268402:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff920a52890000 x1875444569678464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.268413:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569678464/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.268420:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.268423:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573154.268426:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573154.268431:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678592:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573154.268446:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573154.268455:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff920a52892680 x1875444569678592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.268465:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569678592/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.268473:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.268476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573154.268478:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678592:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573154.268483:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573154.268498:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573154.268536:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573154/real 1788573154] req@ffff920a52891500 x1875444569678720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573154.268554:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569678720/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573154.268561:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573154.268564:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573154.268567:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678720:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573154.715751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573155.291707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573155.739762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573156.315644:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573156.763739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573157.339692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573157.787733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573158.363660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573158.811739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573159.387658:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573159.387666:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573159.387670:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573159.387675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573159.387680:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9912 00000100:00080000:1.0:1788573159.387689:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573159.387696:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573159.387730:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569678848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573159.387753:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573159.387756:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573159.387757:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573159.387761:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9912 00000100:00080000:1.0:1788573159.387763:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573159.387766:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573159.387774:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573159.387781:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569678976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573159.387788:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573159.387798:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573159.387800:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573159.387803:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9912 00000100:00080000:1.0:1788573159.387806:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573159.387811:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569678848:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573159.387812:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573159.387820:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573159.387830:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569679104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573159.387838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573159.387863:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573159.387865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573159.387868:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9912 00000100:00080000:1.0:1788573159.387882:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573159.387886:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573159.387890:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573159.387895:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573159.387900:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569679232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573159.387908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573159.387910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573159.387912:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573159.387915:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9912 00000100:00080000:1.0:1788573159.387917:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573159.387920:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573159.387925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573159.387934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569679360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573159.387936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff920a52891c00 x1875444569678848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.387965:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569678848/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.387977:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.387981:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573159.387984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569678848:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573159.387990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678976:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573159.388011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573159.388023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff920a52893800 x1875444569678976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.388034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569678976/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.388040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.388043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573159.388046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569678976:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573159.388066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573159.388079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573159.388086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff920a52890a80 x1875444569679104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.388095:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569679104/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.388100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.388103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573159.388107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573159.388112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679232:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573159.388130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573159.388144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff920a52891f80 x1875444569679232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.388157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569679232/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.388165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.388169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573159.388174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679232:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573159.388183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573159.388203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573159.388214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573159/real 1788573159] req@ffff920a52892a00 x1875444569679360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573159.388223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569679360/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573159.388229:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573159.388232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573159.388235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679360:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573159.835816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573160.411640:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573160.859772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573161.435656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573161.883733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573162.459659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573162.907774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573163.483662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573163.931759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573164.507709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573164.507723:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573164.507731:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573164.507736:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573164.507745:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9917 00000100:00080000:1.0:1788573164.507756:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573164.507764:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573164.507807:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569679488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573164.507827:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573164.507830:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573164.507831:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573164.507834:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9917 00000100:00080000:1.0:1788573164.507837:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573164.507840:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573164.507848:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573164.507859:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569679616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573164.507865:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569679488:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573164.507869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573164.507875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573164.507876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573164.507880:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9917 00000100:00080000:1.0:1788573164.507881:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573164.507883:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573164.507887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573164.507895:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569679744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573164.507906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573164.507909:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573164.507911:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573164.507913:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9917 00000100:00080000:1.0:1788573164.507922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573164.507925:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573164.507928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573164.507935:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569679872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573164.507942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573164.507948:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573164.507952:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573164.507954:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573164.507957:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9917 00000100:00080000:1.0:1788573164.507959:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573164.507962:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573164.507966:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573164.507973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569680000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573164.507984:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff920a52893480 x1875444569679488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573164.508005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569679488/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573164.508017:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573164.508033:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573164.508037:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569679488:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573164.508064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679616:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573164.508092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573164.508107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff920a52890e00 x1875444569679616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573164.508120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569679616/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573164.508128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573164.508132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573164.508136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679616:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573164.508143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573164.508163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573164.508174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff920a52891180 x1875444569679744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573164.508186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569679744/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573164.508194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573164.508198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573164.508201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573164.508207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679872:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573164.508228:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573164.508240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff920a52890700 x1875444569679872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573164.508252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569679872/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573164.508261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573164.508264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573164.508268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569679872:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573164.508274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573164.508304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573164.508316:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573164/real 1788573164] req@ffff920a52893100 x1875444569680000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573164.508328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569680000/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573164.508337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573164.508341:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573164.508344:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680000:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573164.955815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573165.531669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573165.979708:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573166.555698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573167.003789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573167.579667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573168.027735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573168.603697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573169.051788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573169.627682:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573169.627685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573169.627695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573169.627701:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573169.627711:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9923 00000100:00080000:1.0:1788573169.627722:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573169.627730:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573169.627768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569680128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573169.627800:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573169.627805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573169.627807:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573169.627812:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9923 00000100:00080000:1.0:1788573169.627816:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573169.627820:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573169.627831:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573169.627860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569680128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573169.627864:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569680256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573169.627880:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573169.627889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573169.627891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573169.627896:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9923 00000100:00080000:1.0:1788573169.627909:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573169.627914:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573169.627920:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573169.627924:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573169.627943:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569680384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573169.627951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573169/real 1788573169] req@ffff920a52892300 x1875444569680128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573169.627980:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573169.627981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569680128/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573169.627991:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573169.627997:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573169.628009:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9923 00000100:00080000:0.0:1788573169.628012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573169.628015:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573169.628020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788573169.628025:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573169.628034:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573169.628040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569680128:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573169.628052:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569680512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573169.628063:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573169.628068:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573169.628070:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573169.628075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680256:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573169.628079:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9923 00000100:00080000:1.0:1788573169.628086:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573169.628091:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573169.628098:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573169.628111:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569680640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573169.628116:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573169.628139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573169/real 1788573169] req@ffff920a52892d80 x1875444569680256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573169.628156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569680256/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573169.628165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573169.628169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573169.628173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680256:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573169.628180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573169.628201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573169.628213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573169/real 1788573169] req@ffff920a52891880 x1875444569680384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573169.628225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569680384/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573169.628233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573169.628236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573169.628239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573169.628246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680512:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573169.628267:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573169.628278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573169/real 1788573169] req@ffff920a52893100 x1875444569680512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573169.628290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569680512/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573169.628299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573169.628302:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573169.628305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680512:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573169.628321:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573169.628340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573169.628350:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573169/real 1788573169] req@ffff920a52890700 x1875444569680640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573169.628361:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569680640/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573169.628370:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573169.628373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573169.628376:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680640:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573170.075738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573170.651708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573171.099748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573171.675689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573172.123767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573172.699701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573173.147870:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573173.723713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573174.171736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573174.747723:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573174.747736:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788573174.747738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573174.747741:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573174.747757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9928 00000100:00080000:1.0:1788573174.747769:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573174.747777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573174.747830:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569680768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573174.747855:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573174.747858:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573174.747860:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573174.747863:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9928 00000100:00080000:1.0:1788573174.747865:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573174.747868:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573174.747878:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573174.747888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569680896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573174.747897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573174.747900:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573174.747902:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573174.747910:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569680768:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573174.747910:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9928 00000100:00080000:1.0:1788573174.747917:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573174.747924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573174.747929:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573174.747939:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569681024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573174.747951:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573174.747954:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573174.747955:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573174.747959:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9928 00000100:00080000:1.0:1788573174.747961:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573174.747963:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573174.747967:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573174.747975:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569681152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573174.747980:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573174.747983:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573174.747984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573174.747987:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9928 00000100:00080000:1.0:1788573174.747995:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573174.747998:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573174.748004:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573174.748011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573174.748013:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569681280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573174.748063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573174/real 1788573174] req@ffff920a52891180 x1875444569680768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573174.748086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569680768/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573174.748099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573174.748104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573174.748108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569680768:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573174.748135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680896:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573174.748163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573174.748178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573174/real 1788573174] req@ffff920a52890e00 x1875444569680896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573174.748192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569680896/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573174.748201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573174.748205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573174.748209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569680896:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573174.748216:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573174.748236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573174.748248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573174/real 1788573174] req@ffff920a52893480 x1875444569681024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573174.748260:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569681024/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573174.748276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573174.748280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573174.748284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573174.748291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681152:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573174.748333:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573174.748347:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573174/real 1788573174] req@ffff920a52892a00 x1875444569681152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573174.748360:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569681152/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573174.748369:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573174.748373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573174.748376:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681152:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573174.748383:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573174.748405:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573174.748417:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573174/real 1788573174] req@ffff920a52891f80 x1875444569681280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573174.748430:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569681280/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573199 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573174.748437:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573174.748441:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573174.748445:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681280:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573175.195784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573175.771669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573176.219732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573176.795728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573177.243766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573177.819640:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573178.267722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573178.843759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573179.291776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573179.867656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573179.867668:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573179.867674:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573179.867678:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573179.867687:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9933 00000100:00080000:1.0:1788573179.867696:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573179.867714:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573179.867753:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569681408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573179.867781:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573179.867785:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573179.867788:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573179.867793:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9933 00000100:00080000:1.0:1788573179.867798:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573179.867804:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573179.867812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569681408:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573179.867815:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573179.867831:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569681536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573179.867839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573179.867842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573179.867844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573179.867847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9933 00000100:00080000:1.0:1788573179.867859:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573179.867862:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573179.867866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573179.867871:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573179.867876:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891c00 x1875444569681664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573179.867888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573179.867891:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573179.867892:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573179.867896:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9933 00000100:00080000:1.0:1788573179.867899:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573179.867901:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573179.867905:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573179.867906:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573179/real 1788573179] req@ffff920a52890a80 x1875444569681408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573179.867916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569681792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573179.867923:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573179.867928:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569681408/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573179.867928:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573179.867932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573179.867939:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9933 00000100:00080000:1.0:1788573179.867944:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573179.867949:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788573179.867950:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788573179.867952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573179.867956:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573179.867962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569681920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573179.867963:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569681408:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573179.867990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681536:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573179.868008:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573179.868016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573179/real 1788573179] req@ffff920a52893800 x1875444569681536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573179.868027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569681536/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573179.868036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573179.868040:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573179.868044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681536:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573179.868051:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573179.868072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573179.868084:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573179/real 1788573179] req@ffff920a52891c00 x1875444569681664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573179.868093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891c00 x1875444569681664/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573179.868099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573179.868101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573179.868103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573179.868107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681792:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573179.868122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573179.868154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573179/real 1788573179] req@ffff920a52891500 x1875444569681792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573179.868162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569681792/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573179.868168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573179.868170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573179.868172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681792:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573179.868175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573179.868189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573179.868196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573179/real 1788573179] req@ffff920a52892680 x1875444569681920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573179.868203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569681920/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573204 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573179.868208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573179.868209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573179.868211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569681920:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573180.315751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573180.891747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573181.339747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573181.915694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573182.363710:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573182.939690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573183.387731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573183.963727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573184.411684:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573184.987677:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573184.987685:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788573184.987687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573184.987689:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573184.987700:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9938 00000100:00080000:3.0:1788573184.987710:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573184.987716:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573184.987758:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569682048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573184.987784:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573184.987787:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573184.987788:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573184.987792:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9938 00000100:00080000:3.0:1788573184.987796:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573184.987800:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573184.987808:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573184.987814:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569682048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573184.987821:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569682176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573184.987829:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573184.987832:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573184.987833:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573184.987836:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9938 00000100:00080000:3.0:1788573184.987839:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573184.987841:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573184.987844:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573184.987852:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569682304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573184.987859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573184.987862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573184.987863:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573184.987866:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9938 00000100:00080000:3.0:1788573184.987869:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573184.987874:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573184.987876:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573184.987881:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573184.987889:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569682432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573184.987895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573184.987905:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573184/real 1788573184] req@ffff920a529b2a00 x1875444569682048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573184.987907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573184.987910:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573184.987919:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9938 00000100:00080000:3.0:1788573184.987924:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573184.987927:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573184.987930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573184.987931:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569682048/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573184.987941:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569682560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573184.987943:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573184.987966:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573184.987973:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569682048:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573184.988003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682176:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573184.988022:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573184.988031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573184/real 1788573184] req@ffff920a529b3480 x1875444569682176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573184.988044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569682176/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573184.988050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573184.988052:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573184.988054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682176:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573184.988058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573184.988069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573184.988075:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573184/real 1788573184] req@ffff920a529b2300 x1875444569682304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573184.988086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569682304/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573184.988095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573184.988098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573184.988101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573184.988108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682432:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573184.988129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573184.988148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573184/real 1788573184] req@ffff920a529b1c00 x1875444569682432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573184.988156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569682432/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573184.988166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573184.988169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573184.988171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682432:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573184.988175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573184.988189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573184.988195:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573184/real 1788573184] req@ffff920a529b0700 x1875444569682560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573184.988202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569682560/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573209 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573184.988208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573184.988209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573184.988211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682560:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573185.435751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573186.011721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573186.459759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573187.035714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573187.483703:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573188.059668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573188.507853:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573189.083744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573189.531719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573190.107712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573190.107718:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.107724:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573190.107728:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573190.107737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9943 00000100:00080000:1.0:1788573190.107746:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573190.107753:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573190.107798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569682688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573190.107819:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.107822:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573190.107824:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573190.107827:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9943 00000100:00080000:1.0:1788573190.107830:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573190.107834:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573190.107842:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573190.107854:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569682688:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573190.107855:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569682816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573190.107869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.107871:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573190.107872:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573190.107875:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9943 00000100:00080000:1.0:1788573190.107880:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573190.107884:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573190.107889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573190.107903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569682944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573190.107914:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.107920:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788573190.107921:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573190.107924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573190.107931:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9943 00000100:00080000:1.0:1788573190.107936:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573190.107941:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573190.107947:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573190.107966:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569683072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573190.107967:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff920a52890000 x1875444569682688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.107991:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573190.107995:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573190.107998:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573190.108002:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9943 00000100:00100000:0.0:1788573190.108007:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569682688/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573190.108009:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573190.108019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788573190.108025:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788573190.108027:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573190.108031:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573190.108036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569682688:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573190.108045:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569683200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573190.108059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682816:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573190.108082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573190.108096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff920a52893b80 x1875444569682816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573190.108110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569682816/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573190.108119:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573190.108122:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573190.108126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682816:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573190.108133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573190.108151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573190.108160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff920a52890380 x1875444569682944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573190.108168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569682944/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573190.108173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573190.108175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573190.108177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569682944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573190.108182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683072:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573190.108194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573190.108200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff920a52892680 x1875444569683072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573190.108207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569683072/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573190.108212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573190.108214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573190.108216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683072:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573190.108219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573190.108229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573190.108234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573190/real 1788573190] req@ffff920a52891500 x1875444569683200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573190.108241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569683200/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573190.108246:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573190.108248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573190.108250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683200:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573190.555843:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573191.131767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573191.579792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573192.155711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573192.603789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573193.179720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573193.627821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573194.203728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573194.651841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573195.227695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573195.227750:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.227756:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573195.227760:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.227770:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9948 00000100:00080000:3.0:1788573195.227781:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.227791:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573195.227828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569683328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.227859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.227863:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573195.227866:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.227871:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9948 00000100:00080000:3.0:1788573195.227874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.227878:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573195.227887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573195.227898:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569683456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.227906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.227909:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573195.227910:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.227913:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9948 00000100:00080000:3.0:1788573195.227916:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.227918:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573195.227921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573195.227929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569683584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573195.227936:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.227939:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573195.227941:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.227946:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9948 00000100:00080000:3.0:1788573195.227949:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.227953:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573195.227957:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573195.227966:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569683712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573195.227976:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569683328:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573195.227977:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573195.227984:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573195.227987:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573195.227991:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9948 00000100:00080000:3.0:1788573195.227994:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573195.227997:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573195.228001:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573195.228013:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569683840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573195.228048:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573195.228087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573195] req@ffff920a529b2d80 x1875444569683328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573195.228109:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569683328/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573195.228120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573195.228126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573195.228130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569683328:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573195.228139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683456:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573195.228165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573195.228177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573195] req@ffff920a529b3100 x1875444569683456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573195.228191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569683456/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573195.228199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573195.228202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573195.228206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683456:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573195.228212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573195.228230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573195.228241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573195] req@ffff920a529b0000 x1875444569683584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573195.228252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569683584/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573195.228260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573195.228264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573195.228267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573195.228290:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683712:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573195.228310:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573195.228321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573195] req@ffff920a529b1f80 x1875444569683712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573195.228340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569683712/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573195.228349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573195.228352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573195.228356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683712:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573195.228364:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573195.228383:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573195.228395:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573195/real 1788573195] req@ffff920a529b0e00 x1875444569683840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573195.228405:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569683840/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573195.228412:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573195.228416:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573195.228420:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569683840:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573195.675719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573196.251700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573196.699739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573197.275676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573197.723754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573198.299736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573198.747793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573199.323742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573199.771751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573200.347701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573200.347708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573200.347716:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573200.347720:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573200.347728:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9953 00000100:00080000:0.0:1788573200.347737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573200.347743:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573200.347773:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569683968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573200.347794:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573200.347797:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573200.347799:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573200.347802:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9953 00000100:00080000:0.0:1788573200.347805:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573200.347808:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573200.347815:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573200.347824:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569684096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573200.347832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569683968:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573200.347835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573200.347841:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573200.347842:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573200.347846:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9953 00000100:00080000:0.0:1788573200.347848:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573200.347850:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573200.347853:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573200.347862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569684224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573200.347869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573200.347872:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573200.347874:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573200.347876:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9953 00000100:00080000:0.0:1788573200.347878:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573200.347881:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573200.347883:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573200.347892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569684352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573200.347897:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573200.347905:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573200.347908:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573200.347910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573200.347914:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9953 00000100:00080000:0.0:1788573200.347918:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573200.347920:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573200.347923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573200.347934:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff920a48350700 x1875444569683968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573200.347935:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569684480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573200.347981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569683968/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.347992:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.347996:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573200.348001:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569683968:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573200.348025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684096:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573200.348050:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573200.348064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff920a48350380 x1875444569684096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.348078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569684096/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.348084:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.348086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573200.348089:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684096:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573200.348093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573200.348108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573200.348116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff920a48353480 x1875444569684224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.348123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569684224/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.348133:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.348135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573200.348137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573200.348140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684352:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573200.348151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573200.348156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff920a48350a80 x1875444569684352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.348163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569684352/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.348168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.348170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573200.348172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684352:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573200.348175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573200.348185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573200.348190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573200/real 1788573200] req@ffff920a48353b80 x1875444569684480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573200.348197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569684480/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573200.348202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573200.348204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573200.348206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684480:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573200.795751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573201.371673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573201.819842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573202.395724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:3.0:1788573202.843685:0:167376:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 2/2 locks to regular blp list, 1 blwis in pool 00010000:00010000:1.0:1788573202.843735:0:160930:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1788573202.843739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573202.843747:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 2/0,0 mode: PR/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788573202.843787:0:160930:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788573202.843825:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 1/0,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788573202.843831:0:160930:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 1/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788573202.843835:0:160930:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:1.0:1788573202.843866:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a79800/0x8364c713e25bb764 lrc: 0/0,0 mode: --/PR res: [0x200027501:0xa:0x0].0x0 bits 0x11/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c04cc expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1788573202.843885:0:160930:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff920a47b93800 lock: ffff920a60a78200/0x8364c713e25bb75d lrc: 0/0,0 mode: --/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0x265cfe11109c04b0 expref: -99 pid: 167275 timeout: 0 lvb_type: 0 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1788573202.843900:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573202.843903:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1788573202.843905:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573202.843916:0:2397:0:(client.c:1846:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff920a52891c00 x1875444569684608/t0(0) o103->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573203.419697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573203.867759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573204.443698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573204.891728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573205.467706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573205.467715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573205.467724:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573205.467728:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573205.467745:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9958 00000100:00080000:0.0:1788573205.467759:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573205.467766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573205.467801:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569684736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573205.467821:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573205.467824:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573205.467825:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573205.467828:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9958 00000100:00080000:0.0:1788573205.467830:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573205.467841:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573205.467851:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573205.467858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569684736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573205.467868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569684864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573205.467881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573205.467885:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573205.467888:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573205.467893:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9958 00000100:00080000:0.0:1788573205.467897:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573205.467902:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573205.467908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573205.467918:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569684992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573205.467928:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573205.467937:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573205.467941:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573205.467943:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573205.467947:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9958 00000100:00080000:0.0:1788573205.467951:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573205.467953:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573205.467957:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573205.467965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569685120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573205.467972:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff920a48351c00 x1875444569684736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573205.467975:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573205.467982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573205.467984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573205.467992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9958 00000100:00080000:0.0:1788573205.467998:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573205.468002:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573205.468005:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573205.468010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569684736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573205.468015:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569685248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573205.468041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573205.468047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573205.468052:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569684736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573205.468097:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573205.468122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573205.468136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff920a48351500 x1875444569684864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573205.468149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569684864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573205.468158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573205.468161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573205.468165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573205.468174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573205.468195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573205.468207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff920a48353800 x1875444569684992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573205.468219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569684992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573205.468226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573205.468230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573205.468233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569684992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573205.468239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573205.468257:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573205.468267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff920a48353b80 x1875444569685120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573205.468278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569685120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573205.468286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573205.468290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573205.468293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573205.468299:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573205.468318:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573205.468344:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573205/real 1788573205] req@ffff920a48350a80 x1875444569685248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573205.468357:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569685248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573205.468366:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573205.468369:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573205.468373:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573205.915822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573206.491642:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573206.939811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573207.515764:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573207.963751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573208.539702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573208.987740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573209.563748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573210.011729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573210.587679:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.587691:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573210.587696:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573210.587704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573210.587709:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9963 00000100:00080000:1.0:1788573210.587721:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.587729:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573210.587768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569685376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573210.587796:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.587800:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573210.587801:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.587804:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9963 00000100:00080000:1.0:1788573210.587808:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.587811:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573210.587819:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573210.587829:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569685504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573210.587837:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569685376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573210.587839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.587845:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573210.587847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.587851:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9963 00000100:00080000:1.0:1788573210.587853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.587855:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573210.587859:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573210.587866:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569685632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573210.587872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.587875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573210.587876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.587878:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9963 00000100:00080000:1.0:1788573210.587881:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.587883:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573210.587886:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573210.587894:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569685760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573210.587899:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573210.587902:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573210.587903:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573210.587907:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9963 00000100:00080000:1.0:1788573210.587910:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573210.587912:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573210.587916:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573210.587924:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573210.587925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569685888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573210.587958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573210.587974:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573210.587988:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573211.035746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573211.611765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573212.059876:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573212.635696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573213.083786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573213.659707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788573213.659857:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788573213.659894:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573213.659910:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573213.659949:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573213.659957:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573213.659962:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573213.659972:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788573213.659998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573213] req@ffff920a52893480 x1875444569685888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573213.660035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569685888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573213.660054:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573213.660060:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573213.660065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685888:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573213.660073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573213] req@ffff920a52890a80 x1875444569685504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573213.660086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569685504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573213.660096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573213.660101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573213.660104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573213.660112:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573213] req@ffff920a52891f80 x1875444569685632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573213.660124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569685632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573213.660131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573213.660135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573213.660138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573213.660145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573213] req@ffff920a52892a00 x1875444569685760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573213.660157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569685760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573213.660165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573213.660198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573213.660202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569685760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573213.660210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573210/real 1788573213] req@ffff920a52893800 x1875444569685376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573213.660223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569685376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573213.660231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573213.660235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573213.660239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569685376:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788573214.107756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573214.683681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573215.131746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573215.707721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573215.707797:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573215.707805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573215.707810:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573215.707821:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9969 00000100:00080000:3.0:1788573215.707832:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573215.707840:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573215.707885:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569686016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573215.707928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573215.707932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573215.707933:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573215.707936:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9969 00000100:00080000:3.0:1788573215.707939:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573215.707942:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573215.707952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573215.707962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569686144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573215.707970:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573215.707973:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573215.707974:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573215.707978:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9969 00000100:00080000:3.0:1788573215.707982:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573215.707984:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569686016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573215.707987:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573215.707996:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573215.708008:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569686272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573215.708021:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573215.708025:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573215.708027:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573215.708032:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9969 00000100:00080000:3.0:1788573215.708071:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573215.708075:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573215.708081:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573215.708085:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573215.708098:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569686400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573215.708109:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573215.708123:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573215.708125:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573215.708143:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9969 00000100:00080000:3.0:1788573215.708148:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573215.708152:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573215.708158:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573215.708173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573215/real 1788573215] req@ffff920a529b3b80 x1875444569686016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573215.708174:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569686528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573215.708231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569686016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573215.708250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573215.708257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573215.708263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569686016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573215.708274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573215.708316:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573215.708331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573215/real 1788573215] req@ffff920a529b0380 x1875444569686144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573215.708350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569686144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573215.708358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573215.708362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573215.708366:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573215.708415:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573215.708439:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573215.708452:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573215/real 1788573215] req@ffff920a529b1180 x1875444569686272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573215.708465:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569686272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573215.708475:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573215.708479:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573215.708483:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573215.708490:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573215.708514:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573215.708526:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573215/real 1788573215] req@ffff920a529b1500 x1875444569686400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573215.708540:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569686400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573215.708549:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573215.708553:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573215.708556:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573215.708563:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573215.708627:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573215.708641:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573215/real 1788573215] req@ffff920a529b0e00 x1875444569686528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573215.708655:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569686528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573240 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573215.708664:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573215.708668:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573215.708672:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573216.155821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573216.731711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573217.179830:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573217.755712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573218.203783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573218.779688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573219.227799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573219.803721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573220.251752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573220.827707:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573220.827714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573220.827724:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573220.827728:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573220.827740:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9974 00000100:00080000:2.0:1788573220.827753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573220.827761:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573220.827805:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569686656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573220.827831:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573220.827834:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573220.827835:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573220.827839:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9974 00000100:00080000:2.0:1788573220.827841:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573220.827845:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573220.827856:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573220.827866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569686656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573220.827867:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569686784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573220.827881:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573220.827884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573220.827885:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573220.827889:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9974 00000100:00080000:2.0:1788573220.827890:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573220.827893:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573220.827895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573220.827903:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569686912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573220.827911:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573220.827913:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573220.827915:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573220.827917:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9974 00000100:00080000:2.0:1788573220.827920:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573220.827923:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573220.827926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573220.827933:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569687040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573220.827939:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573220.827941:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573220.827943:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573220.827945:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9974 00000100:00080000:2.0:1788573220.827948:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573220.827953:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573220.827954:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573220.827960:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573220.827969:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569687168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573220.828003:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573220/real 1788573220] req@ffff920a60ac4700 x1875444569686656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573220.828021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569686656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573220.828030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573220.828045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573220.828049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569686656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573220.828072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573220.828099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573220.828114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573220/real 1788573220] req@ffff920a60ac7800 x1875444569686784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573220.828128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569686784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573220.828138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573220.828141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573220.828143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573220.828148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573220.828165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573220.828173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573220/real 1788573220] req@ffff920a60ac5500 x1875444569686912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573220.828181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569686912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573220.828186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573220.828188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573220.828190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569686912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573220.828194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573220.828209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573220.828219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573220/real 1788573220] req@ffff920a60ac6300 x1875444569687040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573220.828230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569687040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573220.828239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573220.828242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573220.828246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573220.828253:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573220.828275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573220.828288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573220/real 1788573220] req@ffff920a60ac7100 x1875444569687168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573220.828301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569687168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573245 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573220.828307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573220.828309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573220.828311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573221.275804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573221.851721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573222.299759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573222.875708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573223.323773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573223.899736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573224.347759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573224.923697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573225.371713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573225.947695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573225.947716:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573225.947724:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573225.947728:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573225.947739:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9979 00000100:00080000:1.0:1788573225.947749:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573225.947758:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573225.947791:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569687296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573225.947817:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573225.947824:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573225.947826:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573225.947829:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9979 00000100:00080000:1.0:1788573225.947832:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573225.947836:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573225.947844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573225.947855:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569687424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573225.947861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569687296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573225.947868:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573225.947873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573225.947875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573225.947881:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9979 00000100:00080000:1.0:1788573225.947886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573225.947890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573225.947895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573225.947912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569687552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573225.947921:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573225.947929:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573225.947934:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573225.947949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573225.947955:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9979 00000100:00080000:1.0:1788573225.947960:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573225.947966:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573225.947972:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573225.947977:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573225/real 1788573225] req@ffff920a52890e00 x1875444569687296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573225.947988:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569687680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573225.948002:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573225.948003:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569687296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573225.948007:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573225.948012:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573225.948022:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9979 00000100:00080000:0.0:1788573225.948028:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573225.948029:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573225.948039:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788573225.948039:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573225.948048:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573225.948051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569687296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573225.948064:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569687808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573225.948084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573225.948113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573225.948128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573225/real 1788573225] req@ffff920a52891180 x1875444569687424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573225.948142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569687424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573225.948155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573225.948159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573225.948161:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573225.948166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573225.948182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573225.948190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573225/real 1788573225] req@ffff920a52890700 x1875444569687552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573225.948198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569687552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573225.948203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573225.948205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573225.948207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573225.948211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573225.948222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573225.948227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573225/real 1788573225] req@ffff920a52893100 x1875444569687680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573225.948234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569687680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573225.948239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573225.948241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573225.948244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573225.948248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573225.948258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573225.948264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573225/real 1788573225] req@ffff920a52891880 x1875444569687808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573225.948270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569687808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573250 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573225.948276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573225.948278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573225.948281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569687808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573226.395800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573226.971693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573227.419758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573227.995690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573228.443773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573229.019697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573229.467736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573230.043704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573230.491806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573231.067705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573231.067718:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573231.067726:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573231.067731:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573231.067742:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9984 00000100:00080000:1.0:1788573231.067753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573231.067762:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573231.067800:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569687936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573231.067828:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573231.067833:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573231.067835:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573231.067854:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9984 00000100:00080000:1.0:1788573231.067860:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573231.067866:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573231.067877:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573231.067882:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569687936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573231.067895:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569688064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573231.067916:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573231.067920:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573231.067922:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573231.067927:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9984 00000100:00080000:1.0:1788573231.067931:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573231.067936:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573231.067941:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573231.067942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573231.067961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569688192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573231.067972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573231.067977:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff920a52892d80 x1875444569687936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573231.067978:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573231.067982:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573231.067991:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9984 00000100:00080000:1.0:1788573231.067998:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573231.068003:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573231.068008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569687936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573231.068011:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573231.068032:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569688320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573231.068035:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.068043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573231.068049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569687936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573231.068050:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573231.068057:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573231.068060:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573231.068065:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9984 00000100:00080000:1.0:1788573231.068071:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573231.068074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688064:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573231.068078:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573231.068085:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573231.068099:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569688448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573231.068105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573231.068120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff920a52892300 x1875444569688064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.068131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569688064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.068137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.068140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573231.068143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573231.068149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573231.068195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573231.068204:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff920a52891880 x1875444569688192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.068212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569688192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.068217:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.068219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573231.068222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573231.068227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573231.068240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573231.068248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff920a52893100 x1875444569688320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.068257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569688320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.068263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.068265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573231.068267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573231.068282:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573231.068308:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573231.068319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573231/real 1788573231] req@ffff920a52890700 x1875444569688448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573231.068330:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569688448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573231.068340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573231.068343:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573231.068348:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573231.515782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573232.091758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573232.539732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573233.115664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573233.563727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573234.139682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573234.587703:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573235.163669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573235.611790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573236.187680:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573236.187689:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573236.187693:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573236.187702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573236.187704:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9989 00000100:00080000:3.0:1788573236.187716:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573236.187724:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573236.187775:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569688576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573236.187796:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573236.187799:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573236.187800:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573236.187803:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9989 00000100:00080000:3.0:1788573236.187807:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573236.187811:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573236.187820:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573236.187827:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569688576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573236.187834:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569688704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573236.187848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573236.187850:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573236.187851:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573236.187854:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9989 00000100:00080000:3.0:1788573236.187857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573236.187859:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573236.187862:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573236.187871:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569688832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573236.187878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573236.187881:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573236.187884:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573236.187885:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573236.187890:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9989 00000100:00080000:3.0:1788573236.187893:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573236.187896:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573236.187900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573236.187908:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569688960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573236.187915:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573236.187918:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573236.187919:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573236.187922:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9989 00000100:00080000:3.0:1788573236.187926:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573236.187928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff920a529b1f80 x1875444569688576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573236.187931:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573236.187937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573236.187947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569689088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573236.187962:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569688576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.187972:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.187976:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573236.187981:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569688576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573236.188002:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573236.188021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573236.188030:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff920a529b0000 x1875444569688704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.188039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569688704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.188045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.188048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573236.188050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573236.188054:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573236.188073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573236.188083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff920a529b3100 x1875444569688832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.188096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569688832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.188106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.188109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573236.188113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573236.188117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573236.188134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573236.188142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff920a529b2d80 x1875444569688960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.188150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569688960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.188156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.188158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573236.188160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569688960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573236.188164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573236.188174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573236.188179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573236/real 1788573236] req@ffff920a529b0700 x1875444569689088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573236.188187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569689088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573236.188192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573236.188193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573236.188195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573236.635708:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573237.211691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573237.659806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573238.235752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573238.683830:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573239.259748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573239.707816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573240.283749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573240.731770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573241.307705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573241.307718:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573241.307732:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573241.307737:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573241.307748:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9994 00000100:00080000:3.0:1788573241.307760:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573241.307769:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573241.307823:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569689216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573241.307856:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573241.307860:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573241.307861:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573241.307865:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9994 00000100:00080000:3.0:1788573241.307868:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573241.307873:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573241.307885:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573241.307904:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569689344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573241.307911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569689216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573241.307925:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573241.307931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573241.307932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573241.307937:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9994 00000100:00080000:3.0:1788573241.307941:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573241.307945:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573241.307949:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573241.307960:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569689472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573241.307967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573241.307971:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573241.307973:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573241.307976:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9994 00000100:00080000:3.0:1788573241.307979:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573241.307982:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573241.307986:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573241.307994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569689600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573241.308005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573241.308009:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573241.308015:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573241.308017:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573241.308024:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9994 00000100:00080000:3.0:1788573241.308037:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573241.308044:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573241.308053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff920a529b1c00 x1875444569689216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573241.308055:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573241.308075:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569689728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573241.308087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569689216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.308104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.308111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573241.308116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569689216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573241.308144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573241.308169:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573241.308182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff920a529b2300 x1875444569689344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.308194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569689344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.308203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.308206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573241.308209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573241.308215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573241.308234:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573241.308244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff920a529b3480 x1875444569689472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.308255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569689472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.308270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.308274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573241.308277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573241.308284:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573241.308301:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573241.308311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff920a529b2a00 x1875444569689600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.308333:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569689600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.308342:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.308345:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573241.308348:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573241.308355:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573241.308381:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573241.308392:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573241/real 1788573241] req@ffff920a529b0700 x1875444569689728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573241.308404:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569689728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573241.308412:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573241.308415:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573241.308419:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573241.755792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573242.331756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573242.779882:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573243.355683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573243.803772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573244.379683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573244.827823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573245.403714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573245.851760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573246.427694:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573246.427704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573246.427706:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573246.427714:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.427725:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 9999 00000100:00080000:3.0:1788573246.427736:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.427753:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573246.427788:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569689856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.427811:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.427814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.427816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.427819:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9999 00000100:00080000:3.0:1788573246.427822:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.427825:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573246.427833:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.427842:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569689984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.427849:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.427853:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573246.427856:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573246.427857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569689856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573246.427860:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9999 00000100:00080000:3.0:1788573246.427866:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.427870:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573246.427872:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573246.427880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569690112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.427886:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.427888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573246.427890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.427892:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9999 00000100:00080000:3.0:1788573246.427895:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.427897:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573246.427900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573246.427907:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569690240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573246.427912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573246.427914:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573246.427916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573246.427918:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 9999 00000100:00080000:3.0:1788573246.427921:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573246.427922:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573246.427925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573246.427931:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573246.427938:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569690368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573246.427969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff920a529b2d80 x1875444569689856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.427989:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569689856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.428002:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.428006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573246.428011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569689856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573246.428036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573246.428061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573246.428074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff920a529b3100 x1875444569689984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.428088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569689984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.428098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.428102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573246.428107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569689984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573246.428113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573246.428134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573246.428146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff920a529b0000 x1875444569690112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.428158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569690112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.428167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.428171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573246.428174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573246.428180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573246.428200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573246.428212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff920a529b1f80 x1875444569690240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.428225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569690240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.428233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.428237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573246.428240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573246.428246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573246.428265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573246.428276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573246/real 1788573246] req@ffff920a529b0e00 x1875444569690368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573246.428302:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569690368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573246.428311:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573246.428314:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573246.428318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573246.875771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573247.451707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573247.899778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573248.475731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573248.923748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573249.499684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573249.947767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573250.523694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573250.971748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573251.547683:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573251.547692:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573251.547697:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573251.547706:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10004 00000100:00100000:1.0:1788573251.547709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573251.547715:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573251.547724:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573251.547755:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569690496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573251.547784:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573251.547787:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573251.547788:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573251.547792:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10004 00000100:00080000:0.0:1788573251.547795:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573251.547798:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573251.547805:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573251.547815:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569690624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573251.547824:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573251.547827:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573251.547828:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573251.547834:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569690496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573251.547836:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10004 00000100:00080000:0.0:1788573251.547841:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573251.547846:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573251.547851:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573251.547862:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569690752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573251.547869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573251.547872:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573251.547874:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573251.547876:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10004 00000100:00080000:0.0:1788573251.547879:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573251.547881:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573251.547885:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573251.547894:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573251.547896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569690880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573251.547905:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573251.547908:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573251.547909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573251.547913:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10004 00000100:00080000:0.0:1788573251.547915:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573251.547918:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573251.547921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573251.547935:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569691008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573251.547935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff920a48353480 x1875444569690496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.547972:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569690496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.547980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.547984:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573251.547987:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569690496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573251.548005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573251.548021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573251.548029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff920a48350380 x1875444569690624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.548038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569690624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.548048:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.548051:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573251.548053:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573251.548056:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573251.548067:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573251.548072:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff920a48350700 x1875444569690752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.548097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569690752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.548105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.548109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573251.548113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573251.548119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573251.548140:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573251.548149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff920a48350a80 x1875444569690880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.548158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569690880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.548163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.548165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573251.548168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569690880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573251.548175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573251.548194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573251.548205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573251/real 1788573251] req@ffff920a48353b80 x1875444569691008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573251.548218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569691008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573251.548227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573251.548230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573251.548232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573251.995747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573252.571746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573253.019792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573253.595741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573254.043763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573254.619678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573255.067727:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573255.643708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573256.091797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573256.667710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573256.667761:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573256.667766:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573256.667769:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573256.667780:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10010 00000100:00080000:1.0:1788573256.667790:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573256.667797:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573256.667830:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569691136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573256.667853:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573256.667856:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573256.667857:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573256.667860:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10010 00000100:00080000:1.0:1788573256.667863:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573256.667866:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573256.667874:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573256.667883:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569691264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573256.667891:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573256.667893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573256.667895:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573256.667902:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10010 00000100:00080000:1.0:1788573256.667906:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573256.667909:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573256.667913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573256.667913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569691136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573256.667922:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569691392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573256.667930:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573256.667932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573256.667934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573256.667938:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10010 00000100:00080000:1.0:1788573256.667940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573256.667942:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573256.667945:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573256.667954:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569691520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573256.667959:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573256.667961:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573256.667963:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573256.667965:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10010 00000100:00080000:1.0:1788573256.667968:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573256.667971:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573256.667975:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573256.667984:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569691648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573256.668012:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573256.668060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573256/real 1788573256] req@ffff920a52891180 x1875444569691136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573256.668077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569691136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573256.668087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573256.668091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573256.668095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569691136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573256.668121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573256.668144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573256.668160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573256/real 1788573256] req@ffff920a52890e00 x1875444569691264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573256.668174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569691264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573256.668183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573256.668188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573256.668191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573256.668199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573256.668224:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573256.668233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573256/real 1788573256] req@ffff920a52893800 x1875444569691392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573256.668241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569691392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573256.668247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573256.668250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573256.668252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573256.668256:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573256.668268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573256.668274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573256/real 1788573256] req@ffff920a52892a00 x1875444569691520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573256.668287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569691520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573256.668292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573256.668294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573256.668296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573256.668300:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573256.668310:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573256.668316:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573256/real 1788573256] req@ffff920a52891f80 x1875444569691648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573256.668323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569691648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573281 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573256.668328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573256.668330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573256.668332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573257.115763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573257.691639:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573258.139795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573258.715736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573259.163785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573259.739755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573260.187815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573260.763683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573261.211752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573261.787723:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573261.787735:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573261.787740:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573261.787744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573261.787754:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10015 00000100:00080000:1.0:1788573261.787766:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573261.787775:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573261.787822:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569691776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573261.787845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573261.787848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573261.787850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573261.787853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10015 00000100:00080000:1.0:1788573261.787855:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573261.787858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573261.787867:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573261.787878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569691904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573261.787888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573261.787892:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573261.787894:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573261.787900:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10015 00000100:00080000:1.0:1788573261.787907:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573261.787908:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569691776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573261.787915:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573261.787928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573261.787948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569692032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573261.787967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573261.787972:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573261.787974:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573261.787979:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10015 00000100:00080000:1.0:1788573261.787983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573261.787987:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573261.787993:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573261.787999:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573261.788007:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569692160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573261.788016:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573261.788020:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573261.788023:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573261.788028:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10015 00000100:00080000:1.0:1788573261.788032:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573261.788038:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573261.788045:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573261.788048:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573261/real 1788573261] req@ffff920a52890a80 x1875444569691776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573261.788065:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569692288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573261.788103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569691776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573261.788117:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573261.788123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573261.788127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569691776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573261.788152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573261.788176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573261.788188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573261/real 1788573261] req@ffff920a52893480 x1875444569691904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573261.788200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569691904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573261.788208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573261.788212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573261.788216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569691904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573261.788222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573261.788241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573261.788251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573261/real 1788573261] req@ffff920a52891500 x1875444569692032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573261.788286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569692032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573261.788303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573261.788307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573261.788311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573261.788318:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573261.788340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573261.788351:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573261/real 1788573261] req@ffff920a52892680 x1875444569692160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573261.788363:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569692160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573261.788371:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573261.788374:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573261.788378:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573261.788393:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573261.788412:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573261.788424:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573261/real 1788573261] req@ffff920a52890380 x1875444569692288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573261.788436:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569692288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573261.788444:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573261.788448:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573261.788452:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573262.235737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573262.811665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573263.259762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573263.835702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573264.283816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573264.859744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573265.307779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573265.883747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573266.331754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573266.907704:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573266.907705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573266.907716:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573266.907723:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573266.907737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10020 00000100:00080000:1.0:1788573266.907751:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573266.907761:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573266.907808:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569692416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573266.907834:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573266.907837:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573266.907839:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573266.907843:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10020 00000100:00080000:1.0:1788573266.907846:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573266.907850:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573266.907860:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573266.907867:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569692416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573266.907873:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569692544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573266.907882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573266.907884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573266.907886:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573266.907890:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10020 00000100:00080000:1.0:1788573266.907892:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573266.907894:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573266.907897:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573266.907908:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569692672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573266.907920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573266.907923:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573266.907924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573266.907927:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10020 00000100:00080000:1.0:1788573266.907929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573266.907932:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573266.907935:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573266.907942:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569692800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573266.907947:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573266.907949:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573266.907951:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573266.907954:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573266.907957:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10020 00000100:00080000:1.0:1788573266.907962:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573266.907967:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573266.907972:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573266.907986:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569692928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573266.907997:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573266/real 1788573266] req@ffff920a52893b80 x1875444569692416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573266.908015:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569692416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573266.908025:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573266.908029:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573266.908034:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569692416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573266.908059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573266.908086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573266.908101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573266/real 1788573266] req@ffff920a52890000 x1875444569692544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573266.908124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569692544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573266.908134:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573266.908139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573266.908145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573266.908154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573266.908181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573266.908215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573266/real 1788573266] req@ffff920a52890380 x1875444569692672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573266.908230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569692672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573266.908240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573266.908243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573266.908248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573266.908254:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573266.908283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573266.908301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573266/real 1788573266] req@ffff920a52892680 x1875444569692800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573266.908317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569692800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573266.908327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573266.908332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573266.908336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573266.908344:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573266.908381:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573266.908403:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573266/real 1788573266] req@ffff920a52891500 x1875444569692928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573266.908421:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569692928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573266.908430:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573266.908434:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573266.908439:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569692928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573267.355797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573267.931722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573268.379858:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573268.955731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573269.403806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573269.979706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573270.427800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573271.003733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573271.451758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573272.027730:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573272.027736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573272.027739:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573272.027744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.027753:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10025 00000100:00080000:2.0:1788573272.027762:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.027768:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573272.027800:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569693056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.027820:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.027823:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573272.027824:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.027828:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10025 00000100:00080000:2.0:1788573272.027831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.027835:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573272.027843:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573272.027852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569693056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573272.027854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569693184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.027865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.027868:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573272.027869:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.027872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10025 00000100:00080000:2.0:1788573272.027874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.027876:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573272.027879:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573272.027888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569693312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.027895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.027897:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573272.027899:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.027902:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10025 00000100:00080000:2.0:1788573272.027904:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.027906:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573272.027910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573272.027913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573272.027922:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569693440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573272.027930:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573272.027933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573272.027934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573272.027937:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10025 00000100:00080000:2.0:1788573272.027940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573272.027943:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573272.027946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573272.027948:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff920a60ac5180 x1875444569693056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573272.027956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569693568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573272.027967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569693056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.027977:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.027981:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573272.027984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569693056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573272.028001:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573272.028017:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573272.028034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff920a60ac6a00 x1875444569693184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.028043:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569693184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.028049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.028051:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573272.028053:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573272.028057:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573272.028068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573272.028074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff920a60ac7100 x1875444569693312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.028081:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569693312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.028086:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.028088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573272.028091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573272.028094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573272.028104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573272.028109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff920a60ac6300 x1875444569693440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.028116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569693440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.028121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.028136:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573272.028140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573272.028144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573272.028158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573272.028172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573272/real 1788573272] req@ffff920a60ac5500 x1875444569693568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573272.028179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569693568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573272.028185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573272.028187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573272.028189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573272.475730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573273.051689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573273.499781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573274.075772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573274.523820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573275.099656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573275.547759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573276.123685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573276.571800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573277.147696:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.147704:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573277.147707:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573277.147716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573277.147716:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10030 00000100:00080000:2.0:1788573277.147726:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.147736:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573277.147763:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569693696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.147780:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.147782:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573277.147784:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.147787:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10030 00000100:00080000:2.0:1788573277.147789:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.147792:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573277.147799:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573277.147807:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569693824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.147814:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573277.147816:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569693696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573277.147818:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573277.147822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.147828:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10030 00000100:00080000:2.0:1788573277.147831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.147835:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573277.147839:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573277.147852:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569693952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.147862:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.147865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573277.147867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.147869:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10030 00000100:00080000:2.0:1788573277.147872:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.147874:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573277.147878:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573277.147887:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569694080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573277.147894:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573277.147896:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573277.147898:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573277.147900:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10030 00000100:00080000:2.0:1788573277.147902:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573277.147904:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573277.147908:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693824:192.168.202.157@tcp:38:kworker.0 00000008:00100000:2.0:1788573277.147908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573277.147920:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569694208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573277.147943:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573277.147963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573277.147984:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573277.595795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573278.171768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573278.619785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573279.195795:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573279.643816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573280.219853:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573280.219962:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573280.220017:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573280.220052:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573280.220093:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573280.220104:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573280.220113:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573280.220122:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788573280.220152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573280] req@ffff920a60ac4a80 x1875444569694208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573280.220181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569694208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573280.220200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573280.220205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573280.220210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694208:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573280.220218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573280] req@ffff920a60ac4700 x1875444569693824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573280.220229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569693824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573280.220236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573280.220240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573280.220243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573280.220250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573280] req@ffff920a60ac4380 x1875444569693952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573280.220260:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569693952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573280.220267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573280.220270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573280.220273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569693952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573280.220279:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573280] req@ffff920a60ac4000 x1875444569694080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573280.220288:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569694080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573280.220296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573280.220299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573280.220301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573280.220308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573277/real 1788573280] req@ffff920a60ac7800 x1875444569693696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573280.220320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569693696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573280.220327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573280.220330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573280.220333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569693696:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788573280.667744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573281.243799:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573281.691773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573282.267703:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573282.267717:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573282.267722:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573282.267744:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10035 00000100:00100000:1.0:1788573282.267754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573282.267756:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573282.267765:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573282.267809:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569694336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573282.267841:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573282.267845:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573282.267847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573282.267851:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10035 00000100:00080000:0.0:1788573282.267856:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573282.267860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573282.267873:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573282.267880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569694336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573282.267888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569694464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573282.267899:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573282.267903:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573282.267905:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573282.267910:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10035 00000100:00080000:0.0:1788573282.267914:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573282.267918:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573282.267923:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573282.267934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569694592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573282.267950:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573282.267954:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573282.267955:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573282.267959:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10035 00000100:00080000:0.0:1788573282.267964:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573282.267965:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573282.267973:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573282.267979:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573282.267994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569694720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573282.268023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff920a48353800 x1875444569694336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573282.268032:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573282.268038:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573282.268042:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573282.268050:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10035 00000100:00080000:0.0:1788573282.268054:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573282.268063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569694336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573282.268064:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573282.268074:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573282.268079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573282.268085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573282.268090:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569694848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573282.268092:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569694336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573282.268130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573282.268157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573282.268171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff920a48351500 x1875444569694464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573282.268181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569694464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573282.268189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573282.268192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573282.268194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573282.268200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573282.268216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573282.268226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff920a48351c00 x1875444569694592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573282.268236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569694592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573282.268247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573282.268251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573282.268253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573282.268258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573282.268286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573282.268296:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff920a48353b80 x1875444569694720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573282.268306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569694720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573282.268314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573282.268317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573282.268320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573282.268326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573282.268344:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573282.268354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573282/real 1788573282] req@ffff920a48350a80 x1875444569694848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573282.268366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569694848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573282.268373:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573282.268377:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573282.268381:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569694848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573282.715799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573283.291856:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573283.739975:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573284.315679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573284.763744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573285.339710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573285.787767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573286.363719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573286.811761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573287.387655:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.387667:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573287.387672:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.387683:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10040 00000100:00080000:0.0:1788573287.387694:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.387703:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573287.387744:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569694976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573287.387780:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.387789:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573287.387792:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.387798:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10040 00000100:00080000:0.0:1788573287.387803:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.387809:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573287.387812:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000008:00100000:0.0:1788573287.387822:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573287.387839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569695104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573287.387850:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569694976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573287.387853:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.387860:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573287.387862:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.387868:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10040 00000100:00080000:0.0:1788573287.387871:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.387889:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573287.387895:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573287.387907:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569695232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573287.387916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573287.387919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.387923:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573287.387925:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.387931:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10040 00000100:00080000:0.0:1788573287.387935:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.387949:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573287.387955:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573287.387972:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff920a48350700 x1875444569694976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573287.387973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569695360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573287.387984:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573287.387991:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573287.387993:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573287.388001:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10040 00000100:00080000:0.0:1788573287.388005:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573287.388009:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573287.388014:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573287.388015:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569694976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573287.388026:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569695488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573287.388028:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573287.388037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573287.388045:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569694976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573287.388073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573287.388098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573287.388111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff920a48350380 x1875444569695104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573287.388122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569695104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573287.388129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573287.388133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573287.388136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573287.388141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573287.388162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573287.388173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff920a48353480 x1875444569695232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573287.388184:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569695232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573287.388190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573287.388193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573287.388195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573287.388200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573287.388220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573287.388230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff920a48350a80 x1875444569695360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573287.388239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569695360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573287.388246:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573287.388249:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573287.388252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573287.388258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573287.388274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573287.388283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573287/real 1788573287] req@ffff920a48353b80 x1875444569695488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573287.388298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569695488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573287.388305:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573287.388308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573287.388310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573287.835731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573288.411675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573288.859755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573289.435667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573289.883778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573290.459671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573290.907770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573291.483720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573291.931765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573292.507675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573292.507722:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573292.507731:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573292.507736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573292.507746:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10045 00000100:00080000:3.0:1788573292.507756:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573292.507764:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573292.507807:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569695616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573292.507832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573292.507835:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573292.507836:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573292.507840:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10045 00000100:00080000:3.0:1788573292.507842:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573292.507845:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573292.507855:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573292.507867:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569695744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573292.507875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573292.507893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573292.507895:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573292.507902:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10045 00000100:00100000:2.0:1788573292.507904:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569695616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573292.507907:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573292.507917:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573292.507923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573292.507937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569695872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573292.507945:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573292.507947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573292.507949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573292.507951:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10045 00000100:00080000:3.0:1788573292.507964:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573292.507968:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573292.507971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573292.507974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573292.507986:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569696000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573292.507996:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573292.507998:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573292.508001:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573292.508006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10045 00000100:00080000:3.0:1788573292.508011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573292.508014:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573292.508019:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573292.508025:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff920a529b1500 x1875444569695616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573292.508032:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569696128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573292.508089:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569695616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573292.508101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573292.508105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573292.508108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569695616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573292.508129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573292.508150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573292.508159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff920a529b1180 x1875444569695744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573292.508182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569695744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573292.508188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573292.508190:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573292.508192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573292.508201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573292.508214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573292.508224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff920a529b0380 x1875444569695872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573292.508236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569695872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573292.508245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573292.508248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573292.508252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569695872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573292.508260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573292.508280:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573292.508289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff920a529b3b80 x1875444569696000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573292.508296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569696000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573292.508307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573292.508309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573292.508311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573292.508315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573292.508326:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573292.508331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573292/real 1788573292] req@ffff920a529b0e00 x1875444569696128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573292.508338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569696128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573292.508343:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573292.508345:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573292.508347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573292.955762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573293.531705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573293.979787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573294.555662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573295.003768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573295.579739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573296.027723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573296.603694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573297.051813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573297.627669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573297.627734:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573297.627743:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573297.627748:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573297.627761:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10051 00000100:00080000:1.0:1788573297.627773:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573297.627781:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573297.627847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569696256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573297.627883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573297.627887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573297.627889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573297.627895:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10051 00000100:00080000:1.0:1788573297.627900:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573297.627907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573297.627922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573297.627927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569696256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573297.627940:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569696384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573297.627954:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573297.627957:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573297.627959:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573297.627962:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10051 00000100:00080000:1.0:1788573297.627966:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573297.627969:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573297.627973:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573297.627984:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569696512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573297.627997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573297.628000:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573297.628002:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573297.628006:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10051 00000800:00000100:2.0:1788573297.628014:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573297.628017:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573297.628023:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573297.628027:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573297.628039:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569696640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573297.628048:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573297.628051:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573297.628054:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573297.628060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573297/real 1788573297] req@ffff920a52893480 x1875444569696256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573297.628064:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10051 00000100:00080000:1.0:1788573297.628074:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573297.628089:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573297.628102:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573297.628110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569696256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573297.628115:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569696768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573297.628162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573297.628182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573297.628187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569696256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573297.628218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573297.628246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573297.628263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573297/real 1788573297] req@ffff920a52890a80 x1875444569696384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573297.628277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569696384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573297.628286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573297.628290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573297.628293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573297.628298:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573297.628323:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573297.628331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573297/real 1788573297] req@ffff920a52891f80 x1875444569696512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573297.628339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569696512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573297.628344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573297.628347:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573297.628349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573297.628354:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573297.628365:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573297.628371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573297/real 1788573297] req@ffff920a52892a00 x1875444569696640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573297.628378:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569696640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573297.628384:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573297.628385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573297.628387:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573297.628391:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573297.628402:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573297.628407:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573297/real 1788573297] req@ffff920a52893800 x1875444569696768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573297.628414:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569696768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573297.628419:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573297.628421:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573297.628423:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569696768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573298.075788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573298.651730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573299.099828:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573299.675697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573300.123825:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573300.699772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573301.147827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573301.723695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573302.171833:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573302.747732:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573302.747742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573302.747744:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573302.747750:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573302.747774:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10056 00000100:00080000:1.0:1788573302.747786:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573302.747803:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573302.747851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569696896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573302.747876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573302.747879:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573302.747881:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573302.747884:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10056 00000100:00080000:1.0:1788573302.747887:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573302.747890:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573302.747900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573302.747917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569697024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573302.747925:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569696896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573302.747935:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573302.747941:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573302.747944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573302.747953:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10056 00000100:00080000:1.0:1788573302.747960:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573302.747965:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573302.747971:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573302.747987:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569697152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573302.748001:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573302.748004:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573302.748007:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573302.748012:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10056 00000100:00080000:1.0:1788573302.748018:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573302.748025:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573302.748027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573302.748034:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573302.748046:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569697280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573302.748056:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573302.748060:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573302.748062:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573302.748068:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10056 00000100:00080000:1.0:1788573302.748073:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573302.748080:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573302.748086:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573302.748102:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569697408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573302.748129:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573302/real 1788573302] req@ffff920a52890e00 x1875444569696896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573302.748155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569696896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573302.748169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573302.748175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573302.748181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569696896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573302.748215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573302.748247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573302.748264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573302/real 1788573302] req@ffff920a52891180 x1875444569697024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573302.748281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569697024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573302.748299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573302.748304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573302.748308:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573302.748316:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573302.748342:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573302.748357:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573302/real 1788573302] req@ffff920a52890700 x1875444569697152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573302.748376:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569697152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573302.748401:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573302.748407:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573302.748411:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573302.748419:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573302.748444:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573302.748457:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573302/real 1788573302] req@ffff920a52893100 x1875444569697280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573302.748469:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569697280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573302.748479:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573302.748482:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573302.748485:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573302.748492:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573302.748524:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573302.748537:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573302/real 1788573302] req@ffff920a52891880 x1875444569697408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573302.748550:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569697408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573302.748560:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573302.748563:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573302.748567:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573303.195824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573303.771735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573304.219773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573304.795713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573305.243753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573305.819705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573306.267749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573306.843748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573307.291793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573307.867680:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573307.867680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573307.867691:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573307.867706:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573307.867715:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10061 00000100:00080000:1.0:1788573307.867724:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573307.867730:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573307.867765:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569697536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573307.867787:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573307.867791:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573307.867793:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573307.867798:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10061 00000100:00080000:1.0:1788573307.867802:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573307.867807:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573307.867817:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573307.867832:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569697664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573307.867833:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569697536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573307.867842:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573307.867846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573307.867847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573307.867851:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10061 00000100:00080000:1.0:1788573307.867854:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573307.867858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573307.867863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573307.867872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569697792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573307.867881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573307.867884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573307.867885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573307.867889:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10061 00000100:00080000:1.0:1788573307.867892:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573307.867896:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573307.867898:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573307.867901:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573307.867910:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569697920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573307.867918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573307.867922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573307.867923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573307.867926:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10061 00000100:00080000:1.0:1788573307.867929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573307.867933:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573307/real 1788573307] req@ffff920a52892300 x1875444569697536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573307.867933:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573307.867938:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573307.867948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569698048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573307.867950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569697536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573307.867962:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573307.867966:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573307.867970:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569697536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573307.867994:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573307.868016:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573307.868026:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573307/real 1788573307] req@ffff920a52892d80 x1875444569697664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573307.868035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569697664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573307.868041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573307.868043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573307.868045:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573307.868058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573307.868070:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573307.868077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573307/real 1788573307] req@ffff920a52891880 x1875444569697792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573307.868085:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569697792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573307.868090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573307.868091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573307.868093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573307.868097:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573307.868108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573307.868113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573307/real 1788573307] req@ffff920a52893100 x1875444569697920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573307.868120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569697920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573307.868126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573307.868128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573307.868130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569697920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573307.868133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573307.868143:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573307.868148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573307/real 1788573307] req@ffff920a52890700 x1875444569698048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573307.868154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569698048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573307.868159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573307.868161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573307.868163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573308.315735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573308.891692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573309.339762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573309.915931:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573310.363752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573310.939746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573311.387830:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573311.963725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573312.411814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573312.987728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573312.987740:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573312.987746:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573312.987759:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10066 00000100:00100000:2.0:1788573312.987767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573312.987778:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573312.987790:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573312.987844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569698176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573312.987871:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573312.987874:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573312.987876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573312.987879:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10066 00000100:00080000:1.0:1788573312.987883:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573312.987890:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573312.987900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573312.987911:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569698304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573312.987919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573312.987922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573312.987924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573312.987927:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10066 00000100:00080000:1.0:1788573312.987929:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573312.987933:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573312.987936:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573312.987950:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569698176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573312.987953:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569698432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573312.987967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573312.987969:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573312.987970:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573312.987972:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10066 00000100:00080000:1.0:1788573312.987974:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573312.987976:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573312.987980:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573312.987989:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569698560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573312.987995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573312.987997:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573312.987998:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573312.988000:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10066 00000100:00080000:1.0:1788573312.988003:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573312.988005:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573312.988009:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573312.988018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569698688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573312.988047:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573312.988104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573312/real 1788573312] req@ffff920a52891180 x1875444569698176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573312.988134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569698176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573312.988149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573312.988153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573312.988158:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569698176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573312.988168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573312.988196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573312.988212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573312/real 1788573312] req@ffff920a52890e00 x1875444569698304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573312.988226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569698304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573312.988236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573312.988240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573312.988244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573312.988275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573312.988296:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573312.988309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573312/real 1788573312] req@ffff920a52893800 x1875444569698432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573312.988322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569698432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573312.988331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573312.988335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573312.988338:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573312.988345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573312.988365:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573312.988376:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573312/real 1788573312] req@ffff920a52892a00 x1875444569698560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573312.988389:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569698560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573312.988398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573312.988401:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573312.988405:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573312.988411:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573312.988431:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573312.988442:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573312/real 1788573312] req@ffff920a52891f80 x1875444569698688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573312.988455:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569698688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573337 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573312.988463:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573312.988466:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573312.988469:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573313.435775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573314.011735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573314.459727:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573315.035695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573315.483783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573316.059753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573316.507766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573317.083690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573317.531735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573318.107731:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.107740:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573318.107746:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573318.107746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573318.107757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10071 00000100:00080000:1.0:1788573318.107768:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.107776:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573318.107809:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569698816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.107832:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.107835:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573318.107836:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.107839:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10071 00000100:00080000:1.0:1788573318.107842:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.107844:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573318.107851:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573318.107861:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569698944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.107868:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.107870:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573318.107872:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.107875:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10071 00000100:00080000:1.0:1788573318.107877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.107883:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573318.107886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569698816:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573318.107887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573318.107896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569699072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.107903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.107905:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573318.107906:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.107909:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10071 00000100:00080000:1.0:1788573318.107911:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.107913:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573318.107916:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573318.107925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569699200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573318.107931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573318.107933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573318.107935:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573318.107938:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10071 00000100:00080000:1.0:1788573318.107948:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573318.107950:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573318.107953:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573318.107961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573318.107964:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569699328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573318.107996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff920a52890a80 x1875444569698816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.108010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569698816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.108022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.108024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573318.108027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569698816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573318.108033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573318.108049:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573318.108057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff920a52893480 x1875444569698944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.108065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569698944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.108070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.108072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573318.108074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569698944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573318.108092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573318.108104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573318.108110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff920a52891500 x1875444569699072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.108117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569699072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.108122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.108124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573318.108126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573318.108131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573318.108150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573318.108161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff920a52892680 x1875444569699200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.108175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569699200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.108183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.108187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573318.108189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573318.108194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573318.108215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573318.108223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573318/real 1788573318] req@ffff920a52890380 x1875444569699328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573318.108230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569699328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573318.108235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573318.108237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573318.108239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573318.555802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573319.131729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573319.579725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573320.155758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573320.603758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573321.179713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573321.627856:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573322.203773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573322.651835:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573323.227749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573323.227755:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.227767:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573323.227773:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.227786:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10076 00000100:00080000:2.0:1788573323.227797:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.227805:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573323.227846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569699456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.227882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.227888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573323.227890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.227895:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10076 00000100:00080000:2.0:1788573323.227899:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.227903:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573323.227923:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573323.227938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569699456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573323.227943:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569699584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.227957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.227961:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573323.227963:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.227968:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10076 00000100:00080000:2.0:1788573323.227973:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573323.227978:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573323.227984:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573323.228003:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573323.228009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569699712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.228021:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.228025:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573323.228028:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573323.228033:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10076 00000100:00080000:2.0:1788573323.228037:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573323.228040:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff920a60ac4e00 x1875444569699456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573323.228045:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573323.228062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569699456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573323.228067:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573323.228072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.228076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573323.228080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569699456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573323.228087:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569699840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573323.228100:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573323.228104:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573323.228106:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573323.228111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699584:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788573323.228113:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10076 00000100:00080000:2.0:1788573323.228121:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573323.228135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573323.228136:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573323.228144:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573323.228146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff920a60ac5c00 x1875444569699584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.228157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569699584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573323.228164:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569699968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573323.228166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.228173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573323.228179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573323.228191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573323.228213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573323.228227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff920a60ac7800 x1875444569699712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.228241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569699712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.228254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.228256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573323.228259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573323.228263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573323.228278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573323.228286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff920a60ac4000 x1875444569699840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.228293:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569699840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.228298:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.228300:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573323.228302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573323.228314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573323.228339:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573323.228346:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573323/real 1788573323] req@ffff920a60ac4380 x1875444569699968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573323.228353:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569699968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573323.228358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573323.228360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573323.228363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569699968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573323.675807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573324.251742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573324.699766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573325.275725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573325.723795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573326.299721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573326.747789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573327.323708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573327.771750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573328.347745:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573328.347755:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573328.347760:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573328.347772:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10081 00000100:00100000:2.0:1788573328.347778:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573328.347784:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573328.347795:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573328.347835:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569700096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573328.347861:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573328.347864:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573328.347865:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573328.347869:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10081 00000100:00080000:3.0:1788573328.347872:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573328.347875:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573328.347882:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573328.347893:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569700224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573328.347901:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573328.347904:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573328.347905:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573328.347908:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10081 00000100:00080000:3.0:1788573328.347910:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573328.347914:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569700096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573328.347915:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573328.347924:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573328.347933:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569700352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573328.347939:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573328.347941:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573328.347943:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573328.347953:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10081 00000100:00080000:3.0:1788573328.347956:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573328.347958:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573328.347963:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573328.347971:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569700480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573328.347977:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573328.347979:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573328.347981:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573328.347983:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10081 00000100:00080000:3.0:1788573328.347986:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573328.347988:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573328.347992:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573328.347996:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573328.348005:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569700608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573328.348032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573328] req@ffff920a529b1f80 x1875444569700096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573328.348055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569700096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573328.348067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573328.348072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573328.348077:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569700096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573328.348085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573328.348111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573328.348125:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573328] req@ffff920a529b0000 x1875444569700224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573328.348139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569700224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573328.348148:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573328.348151:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573328.348155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573328.348181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573328.348205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573328.348218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573328] req@ffff920a529b3100 x1875444569700352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573328.348229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569700352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573328.348235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573328.348237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573328.348239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573328.348243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573328.348259:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573328.348265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573328] req@ffff920a529b2d80 x1875444569700480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573328.348273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569700480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573328.348278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573328.348280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573328.348282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573328.348285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573328.348295:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573328.348301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573328/real 1788573328] req@ffff920a529b0700 x1875444569700608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573328.348308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569700608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573328.348314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573328.348316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573328.348318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573328.795780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573329.371756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573329.819748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573330.395750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573330.843808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573331.419738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573331.867764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573332.443708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573332.891792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573333.467668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573333.467697:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573333.467705:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573333.467711:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573333.467722:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10086 00000100:00080000:3.0:1788573333.467732:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573333.467741:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573333.467777:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569700736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573333.467811:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573333.467815:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573333.467817:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573333.467822:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10086 00000100:00080000:3.0:1788573333.467825:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573333.467830:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573333.467841:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573333.467854:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569700864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573333.467864:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569700736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573333.467867:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573333.467875:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573333.467877:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573333.467883:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10086 00000100:00080000:3.0:1788573333.467887:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573333.467891:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573333.467897:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573333.467910:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569700992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573333.467925:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573333.467930:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573333.467932:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573333.467936:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10086 00000100:00080000:3.0:1788573333.467941:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573333.467942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573333.467950:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573333.467957:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573333.467969:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569701120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573333.467980:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573333.467986:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff920a529b2a00 x1875444569700736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.467986:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573333.467989:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573333.467998:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10086 00000100:00080000:3.0:1788573333.468012:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573333.468019:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569700736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573333.468020:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573333.468029:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788573333.468038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573333.468045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573333.468048:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569701248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573333.468054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569700736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573333.468087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573333.468113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573333.468126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff920a529b3480 x1875444569700864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573333.468139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569700864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573333.468148:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573333.468151:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573333.468155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573333.468161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573333.468187:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573333.468197:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff920a529b2300 x1875444569700992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573333.468209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569700992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573333.468218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573333.468221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573333.468224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569700992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573333.468231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573333.468248:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573333.468258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff920a529b1c00 x1875444569701120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573333.468270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569701120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573333.468278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573333.468281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573333.468285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573333.468291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573333.468309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573333.468319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573333/real 1788573333] req@ffff920a529b0700 x1875444569701248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573333.468331:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569701248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573333.468340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573333.468343:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573333.468346:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573333.915827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573334.491766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573334.939792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573335.515730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573335.963790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573336.539707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573336.987735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573337.563754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573338.011828:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573338.587682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573338.587682:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573338.587693:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573338.587706:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573338.587719:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10091 00000100:00080000:3.0:1788573338.587732:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573338.587740:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573338.587791:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569701376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573338.587821:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573338.587824:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573338.587826:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573338.587829:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10091 00000100:00080000:3.0:1788573338.587832:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573338.587837:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573338.587849:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573338.587862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569701376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573338.587870:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569701504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573338.587885:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573338.587889:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573338.587892:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573338.587896:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10091 00000100:00080000:3.0:1788573338.587899:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573338.587903:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573338.587907:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573338.587919:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569701632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573338.587926:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573338.587929:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573338.587931:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573338.587934:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10091 00000100:00080000:3.0:1788573338.587937:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573338.587940:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573338.587943:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573338.587950:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573338.587953:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569701760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573338.587961:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573338.587964:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573338.587965:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573338.587968:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10091 00000100:00080000:3.0:1788573338.587970:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573338.587973:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573338.587975:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573338.587982:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569701888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573338.588000:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff920a529b2d80 x1875444569701376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573338.588019:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569701376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573338.588028:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573338.588032:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573338.588036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569701376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573338.588062:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573338.588084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573338.588094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff920a529b3100 x1875444569701504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573338.588104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569701504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573338.588115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573338.588118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573338.588121:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573338.588124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573338.588138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573338.588145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff920a529b0000 x1875444569701632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573338.588153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569701632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573338.588158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573338.588160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573338.588162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573338.588166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573338.588180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573338.588187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff920a529b1f80 x1875444569701760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573338.588200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569701760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573338.588210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573338.588214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573338.588219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573338.588228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573338.588260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573338.588276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573338/real 1788573338] req@ffff920a529b0e00 x1875444569701888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573338.588290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569701888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573338.588301:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573338.588305:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573338.588310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569701888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573339.035809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573339.611733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573340.059770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573340.635687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573341.083780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573341.659758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573342.107779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573342.683704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573343.131816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573343.707717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573343.707735:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573343.707745:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573343.707751:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573343.707765:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10097 00000100:00080000:0.0:1788573343.707779:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573343.707792:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573343.707846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569702016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573343.707876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573343.707880:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573343.707881:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573343.707884:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10097 00000100:00080000:0.0:1788573343.707887:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573343.707892:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573343.707902:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573343.707913:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569702144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573343.707920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573343.707923:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573343.707924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573343.707928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10097 00000100:00080000:0.0:1788573343.707929:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573343.707932:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573343.707934:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573343.707943:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569702272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573343.707945:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569702016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573343.707954:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573343.707958:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573343.707959:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573343.707961:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10097 00000100:00080000:0.0:1788573343.707963:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573343.707965:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573343.707968:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573343.707977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569702400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573343.707983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573343.707986:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573343.707987:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573343.707990:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10097 00000100:00080000:0.0:1788573343.707993:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573343.707995:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573343.707998:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573343.708005:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569702528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573343.708070:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573343.708121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573343.708150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573343.708180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573344.155757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573344.731798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573345.179795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573345.755802:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573346.203734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573346.779876:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573346.779995:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573346.780062:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573346.780084:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573346.780143:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573346.780155:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573346.780175:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573346.780186:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788573346.780218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573343/real 1788573346] req@ffff920a48353800 x1875444569702272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573346.780247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569702272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573346.780262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573346.780280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573346.780286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573346.780295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573343/real 1788573346] req@ffff920a48353b80 x1875444569702400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573346.780311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569702400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573346.780320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573346.780324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573346.780328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573346.780336:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573343/real 1788573346] req@ffff920a48350a80 x1875444569702528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573346.780349:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569702528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573346.780356:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573346.780360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573346.780363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702528:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573346.780371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573343/real 1788573346] req@ffff920a48351c00 x1875444569702016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573346.780384:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569702016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573346.780393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573346.780396:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573346.780400:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569702016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573346.780407:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573343/real 1788573346] req@ffff920a48351500 x1875444569702144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573346.780419:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569702144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573346.780428:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573346.780432:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573346.780436:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702144:192.168.202.157@tcp:38:kworker.0 00010000:00010000:2.0:1788573347.227733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573347.803785:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573348.251745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573348.827735:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573348.827746:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573348.827751:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573348.827763:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10102 00000100:00100000:1.0:1788573348.827764:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573348.827774:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573348.827784:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573348.827820:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569702656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573348.827848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573348.827851:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573348.827853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573348.827856:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10102 00000100:00080000:0.0:1788573348.827859:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573348.827862:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573348.827872:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573348.827886:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569702784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573348.827894:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569702656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573348.827902:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573348.827907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573348.827909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573348.827915:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10102 00000100:00080000:0.0:1788573348.827919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573348.827923:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573348.827930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573348.827944:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569702912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573348.827961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573348.827965:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573348.827974:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573348.827976:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573348.827982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10102 00000100:00080000:0.0:1788573348.827986:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573348.827990:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573348.827997:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573348.828010:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573348/real 1788573348] req@ffff920a48351500 x1875444569702656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573348.828012:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569703040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573348.828044:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573348.828048:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788573348.828048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569702656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573348.828050:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573348.828060:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10102 00000100:00080000:0.0:1788573348.828066:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573348.828071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573348.828072:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573348.828082:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573348.828083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573348.828091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569702656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573348.828099:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569703168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573348.828133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573348.828162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573348.828178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573348/real 1788573348] req@ffff920a48351c00 x1875444569702784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573348.828192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569702784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573348.828201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573348.828205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573348.828209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573348.828216:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573348.828238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573348.828250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573348/real 1788573348] req@ffff920a48350a80 x1875444569702912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573348.828263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569702912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573348.828272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573348.828275:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573348.828278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569702912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573348.828286:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573348.828308:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573348.828320:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573348/real 1788573348] req@ffff920a48353b80 x1875444569703040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573348.828333:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569703040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573348.828341:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573348.828344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573348.828347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573348.828354:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573348.828375:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573348.828388:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573348/real 1788573348] req@ffff920a48353800 x1875444569703168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573348.828400:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569703168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573348.828408:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573348.828411:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573348.828414:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573349.275718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573349.851848:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573350.299753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573350.875667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573351.323771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573351.899707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573352.347769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573352.923713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573353.371757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573353.947670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573353.947683:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573353.947690:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573353.947694:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573353.947702:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10107 00000100:00080000:0.0:1788573353.947711:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573353.947718:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573353.947751:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569703296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573353.947775:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573353.947778:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573353.947779:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573353.947782:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10107 00000100:00080000:0.0:1788573353.947784:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573353.947787:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573353.947794:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573353.947802:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569703424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573353.947810:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573353.947813:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573353.947814:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573353.947817:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10107 00000100:00080000:0.0:1788573353.947819:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573353.947821:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573353.947825:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573353.947831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569703296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573353.947832:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569703552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573353.947850:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573353.947853:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573353.947854:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573353.947857:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10107 00000100:00080000:0.0:1788573353.947859:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573353.947862:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573353.947866:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573353.947874:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569703680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573353.947881:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573353.947883:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573353.947885:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573353.947887:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10107 00000100:00080000:0.0:1788573353.947890:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573353.947896:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788573353.947896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573353.947905:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573353.947918:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569703808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573353.947928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573353/real 1788573353] req@ffff920a48353480 x1875444569703296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573353.947944:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569703296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573353.947952:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573353.947955:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573353.947958:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569703296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573353.947963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573353.947977:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573353.947985:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573353/real 1788573353] req@ffff920a48350380 x1875444569703424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573353.947992:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569703424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573353.947998:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573353.948000:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573353.948001:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573353.948019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573353.948029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573353.948035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573353/real 1788573353] req@ffff920a48350700 x1875444569703552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573353.948047:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569703552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573353.948052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573353.948054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573353.948056:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573353.948059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573353.948068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573353.948074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573353/real 1788573353] req@ffff920a48353800 x1875444569703680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573353.948080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569703680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573353.948085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573353.948087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573353.948089:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573353.948092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573353.948102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573353.948107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573353/real 1788573353] req@ffff920a48353b80 x1875444569703808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573353.948114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569703808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573378 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573353.948124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573353.948126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573353.948128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569703808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573354.395724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573354.971713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573355.419743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573355.995666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573356.443707:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573357.019721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573357.467810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573358.043727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573358.491825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573359.067723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573359.067737:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.067748:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573359.067753:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.067763:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10112 00000100:00080000:2.0:1788573359.067773:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.067781:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573359.067815:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569703936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.067837:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.067841:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573359.067842:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.067846:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10112 00000100:00080000:2.0:1788573359.067848:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.067851:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573359.067859:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573359.067870:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569704064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.067878:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.067881:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573359.067883:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.067886:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10112 00000100:00080000:2.0:1788573359.067888:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.067890:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573359.067894:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573359.067900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569703936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573359.067904:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569704192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.067913:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.067921:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573359.067924:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.067927:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10112 00000100:00080000:2.0:1788573359.067929:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.067931:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573359.067935:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573359.067945:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569704320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573359.067951:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573359.067954:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573359.067955:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573359.067957:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10112 00000100:00080000:2.0:1788573359.067959:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573359.067961:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573359.067965:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573359.067971:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569704448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573359.068009:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573359.068054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff920a60ac4700 x1875444569703936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573359.068082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569703936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573359.068095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573359.068100:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573359.068105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569703936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573359.068114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573359.068137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573359.068150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff920a60ac4a80 x1875444569704064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573359.068162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569704064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573359.068171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573359.068175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573359.068178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573359.068206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573359.068226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573359.068236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff920a60ac5500 x1875444569704192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573359.068248:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569704192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573359.068263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573359.068266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573359.068270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573359.068276:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573359.068294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573359.068305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff920a60ac6300 x1875444569704320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573359.068316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569704320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573359.068324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573359.068328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573359.068332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573359.068338:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573359.068356:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573359.068366:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573359/real 1788573359] req@ffff920a60ac7100 x1875444569704448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573359.068377:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569704448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573359.068386:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573359.068389:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573359.068393:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573359.515829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573360.091724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573360.539822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573361.115753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573361.563800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573362.139695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573362.587765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573363.163689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573363.611801:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573364.187739:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573364.187739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573364.187752:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573364.187757:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.187768:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10117 00000100:00080000:0.0:1788573364.187778:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.187786:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573364.187825:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569704576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.187858:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.187862:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573364.187865:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.187869:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10117 00000100:00080000:0.0:1788573364.187873:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.187878:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573364.187889:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573364.187906:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569704704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573364.187920:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569704576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573364.187925:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.187939:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573364.187941:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.187947:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10117 00000100:00080000:0.0:1788573364.187953:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.187958:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573364.187963:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573364.187974:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569704832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.187984:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573364.187987:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573364.187988:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.187991:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10117 00000100:00080000:0.0:1788573364.187994:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.187996:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573364.188001:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573364.188010:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569704960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573364.188017:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573364.188021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573364.188023:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573364.188027:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573364.188031:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10117 00000100:00080000:0.0:1788573364.188036:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573364.188038:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573364.188042:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573364.188061:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569705088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573364.188094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff920a48350a80 x1875444569704576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573364.188117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569704576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573364.188127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573364.188131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573364.188134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569704576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573364.188159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573364.188179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573364.188188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff920a48351c00 x1875444569704704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573364.188197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569704704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573364.188203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573364.188206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573364.188209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573364.188213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573364.188225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573364.188230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff920a48351500 x1875444569704832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573364.188237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569704832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573364.188243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573364.188245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573364.188247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573364.188250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573364.188262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573364.188267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff920a48353b80 x1875444569704960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573364.188275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569704960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573364.188280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573364.188282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573364.188284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569704960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573364.188287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573364.188297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573364.188303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573364/real 1788573364] req@ffff920a48353800 x1875444569705088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573364.188311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569705088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573364.188316:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573364.188318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573364.188320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573364.635793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573365.211721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573365.659802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573366.235662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573366.683733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573367.259684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573367.707791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573368.283673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573368.731738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573369.307687:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.307697:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573369.307703:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.307715:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10122 00000100:00100000:1.0:1788573369.307719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573369.307728:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.307738:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573369.307776:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569705216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.307803:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.307808:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573369.307810:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.307815:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10122 00000100:00080000:0.0:1788573369.307819:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.307824:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573369.307834:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573369.307836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569705216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573369.307859:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569705344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.307871:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.307875:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573369.307877:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.307882:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10122 00000100:00080000:0.0:1788573369.307886:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.307890:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573369.307895:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573369.307907:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569705472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573369.307908:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573369.307931:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573369.307936:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573369.307938:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573369.307943:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10122 00000100:00080000:0.0:1788573369.307947:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.307951:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573369.307954:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff920a48350700 x1875444569705216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573369.307958:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573369.307977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569705216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573369.307978:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569705600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573369.308002:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573369.308005:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573369.308006:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573369.308011:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573369.308011:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573369.308019:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569705216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573369.308021:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10122 00000100:00080000:0.0:1788573369.308027:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573369.308032:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573369.308036:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573369.308044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573369.308049:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569705728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573369.308068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573369.308080:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff920a48350380 x1875444569705344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573369.308093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569705344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573369.308099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573369.308101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573369.308104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573369.308107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573369.308118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573369.308124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff920a48353480 x1875444569705472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573369.308131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569705472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573369.308136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573369.308138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573369.308140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573369.308143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573369.308155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573369.308161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff920a48353800 x1875444569705600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573369.308168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569705600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573369.308177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573369.308180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573369.308182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573369.308191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573369.308201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573369.308206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573369/real 1788573369] req@ffff920a48353b80 x1875444569705728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573369.308213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569705728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573369.308219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573369.308221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573369.308222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573369.755741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573370.331749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573370.779825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573371.355738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573371.803721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573372.379693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573372.827792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573373.403730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573373.851785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573374.427673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573374.427726:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573374.427731:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573374.427735:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573374.427747:0:103578:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10127 00000100:00080000:1.0:1788573374.427759:0:103578:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573374.427768:0:103578:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573374.427808:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569705856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573374.427832:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573374.427835:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573374.427836:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573374.427840:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10127 00000100:00080000:1.0:1788573374.427843:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573374.427847:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573374.427858:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573374.427872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569705856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573374.427873:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569705984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573374.427888:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573374.427891:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573374.427893:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573374.427896:0:103578:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10127 00000100:00080000:1.0:1788573374.427901:0:103578:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573374.427904:0:103578:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573374.427907:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573374.427920:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569706112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573374.427932:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573374.427936:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573374.427939:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573374.427946:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10127 00000100:00080000:1.0:1788573374.427952:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573374.427959:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573374.427960:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573374.427971:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573374.427986:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569706240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573374.427994:0:103578:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573374.427999:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff920a52890000 x1875444569705856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573374.428000:0:103578:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573374.428003:0:103578:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573374.428011:0:103578:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10127 00000100:00080000:1.0:1788573374.428016:0:103578:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573374.428021:0:103578:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573374.428026:0:103578:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573374.428028:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569705856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573374.428039:0:103578:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569706368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573374.428047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573374.428053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573374.428058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569705856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573374.428085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573374.428111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573374.428126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff920a52893b80 x1875444569705984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573374.428140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569705984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573374.428148:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573374.428150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573374.428153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569705984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573374.428157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573374.428175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573374.428185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff920a52890380 x1875444569706112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573374.428193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569706112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573374.428199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573374.428201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573374.428202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573374.428207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573374.428235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573374.428242:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff920a52892680 x1875444569706240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573374.428252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569706240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573374.428258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573374.428260:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573374.428262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573374.428267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573374.428280:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573374.428288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573374/real 1788573374] req@ffff920a52891500 x1875444569706368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573374.428296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569706368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573374.428301:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573374.428304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573374.428307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573374.875792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573375.451680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573375.899785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573376.475770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573376.923835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573377.499689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573377.947780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573378.523715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573378.971773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573379.547695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573379.547728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573379.547734:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573379.547739:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573379.547747:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10132 00000100:00080000:0.0:1788573379.547757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573379.547764:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573379.547797:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569706496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573379.547819:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573379.547822:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573379.547824:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573379.547827:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10132 00000100:00080000:0.0:1788573379.547829:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573379.547831:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573379.547839:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573379.547850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569706624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573379.547859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573379.547862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573379.547863:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573379.547870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10132 00000100:00080000:0.0:1788573379.547873:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573379.547882:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573379.547883:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569706496:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573379.547886:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573379.547901:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569706752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573379.547912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573379.547916:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573379.547920:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573379.547924:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10132 00000100:00080000:0.0:1788573379.547929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573379.547933:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573379.547940:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573379.547954:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569706880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573379.547962:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573379.547965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573379.547968:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573379.547972:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10132 00000100:00080000:0.0:1788573379.547977:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573379.547981:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788573379.547989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573379.547991:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573379.548009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569707008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573379.548050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff920a48351500 x1875444569706496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573379.548075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569706496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573379.548089:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573379.548094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573379.548099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569706496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573379.548132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573379.548160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573379.548175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff920a48351c00 x1875444569706624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573379.548189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569706624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573379.548199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573379.548203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573379.548207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573379.548213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573379.548235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573379.548246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff920a48350a80 x1875444569706752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573379.548265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569706752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573379.548274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573379.548277:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573379.548281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573379.548287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573379.548309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573379.548322:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff920a48353b80 x1875444569706880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573379.548335:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569706880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573379.548344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573379.548348:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573379.548352:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569706880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573379.548359:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573379.548393:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573379.548406:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573379/real 1788573379] req@ffff920a48353800 x1875444569707008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573379.548419:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569707008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573379.548467:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573379.548471:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573379.548484:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573379.995811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573380.571703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573381.019754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573381.595714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573382.043772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573382.619731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573383.067848:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573383.643724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573384.091832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573384.667697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573384.667716:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573384.667725:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573384.667730:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573384.667743:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10138 00000100:00080000:0.0:1788573384.667755:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573384.667763:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573384.667808:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569707136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573384.667831:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573384.667834:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573384.667835:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573384.667838:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10138 00000100:00080000:0.0:1788573384.667841:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573384.667844:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573384.667858:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573384.667867:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569707136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573384.667874:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569707264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573384.667887:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573384.667891:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573384.667893:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573384.667898:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10138 00000100:00080000:0.0:1788573384.667902:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573384.667907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573384.667913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573384.667926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569707392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573384.667936:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573384.667939:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573384.667941:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573384.667945:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573384.667949:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10138 00000100:00080000:0.0:1788573384.667953:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573384.667957:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573384.667963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573384.667973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569707520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573384.667980:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573384.667983:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573384.667985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573384.667989:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573384/real 1788573384] req@ffff920a48353480 x1875444569707136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573384.667990:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10138 00000100:00080000:0.0:1788573384.667995:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573384.668000:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573384.668006:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573384.668018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569707648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573384.668018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569707136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573384.668049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573384.668055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573384.668059:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569707136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573384.668083:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573384.668100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573384.668110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573384/real 1788573384] req@ffff920a48350380 x1875444569707264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573384.668119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569707264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573384.668131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573384.668133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573384.668135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573384.668139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573384.668163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573384.668176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573384/real 1788573384] req@ffff920a48350700 x1875444569707392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573384.668189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569707392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573384.668199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573384.668202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573384.668205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573384.668209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573384.668227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573384.668235:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573384/real 1788573384] req@ffff920a48353800 x1875444569707520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573384.668243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569707520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573384.668249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573384.668251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573384.668254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573384.668258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573384.668281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573384.668293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573384/real 1788573384] req@ffff920a48353b80 x1875444569707648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573384.668305:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569707648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573384.668314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573384.668318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573384.668321:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573385.115772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573385.691758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573386.139740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573386.715703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573387.163805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573387.739675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573388.187788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573388.763728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573389.211774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573389.787671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573389.787678:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573389.787702:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573389.787709:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573389.787722:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10143 00000100:00080000:0.0:1788573389.787735:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573389.787746:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573389.787799:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569707776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573389.787830:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573389.787835:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573389.787838:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573389.787843:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10143 00000100:00080000:0.0:1788573389.787848:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573389.787854:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573389.787860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569707776:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573389.787869:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573389.787888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569707904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573389.787902:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573389.787907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573389.787909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573389.787914:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10143 00000100:00080000:0.0:1788573389.787919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573389.787924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573389.787930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573389.787935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573389.787947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569708032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573389.787960:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573389.787964:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573389.787967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573389.787972:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10143 00000100:00100000:2.0:1788573389.787975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573389/real 1788573389] req@ffff920a48350a80 x1875444569707776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573389.787981:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573389.787990:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573389.787996:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569707776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573389.787998:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573389.788013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573389.788017:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573389.788021:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569708160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573389.788023:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569707776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573389.788041:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573389.788045:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573389.788047:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573389.788052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707904:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573389.788054:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10143 00000100:00080000:0.0:1788573389.788062:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573389.788068:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573389.788075:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573389.788076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573389.788088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573389/real 1788573389] req@ffff920a48351c00 x1875444569707904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573389.788095:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569708288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573389.788102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569707904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573389.788121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573389.788126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573389.788129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569707904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573389.788134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573389.788146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573389.788154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573389/real 1788573389] req@ffff920a48351500 x1875444569708032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573389.788162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569708032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573389.788168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573389.788169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573389.788171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573389.788174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573389.788185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573389.788191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573389/real 1788573389] req@ffff920a48353b80 x1875444569708160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573389.788203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569708160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573389.788208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573389.788209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573389.788211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573389.788221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573389.788232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573389.788237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573389/real 1788573389] req@ffff920a48353800 x1875444569708288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573389.788244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569708288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573389.788248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573389.788250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573389.788252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573390.235771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573390.811731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573391.259759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573391.835716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573392.283798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573392.859702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573393.307755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573393.883710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573394.331807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573394.907672:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573394.907673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573394.907683:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573394.907688:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573394.907698:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10148 00000100:00080000:0.0:1788573394.907709:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573394.907716:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573394.907766:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569708416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573394.907790:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573394.907792:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573394.907794:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573394.907797:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10148 00000100:00080000:0.0:1788573394.907800:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573394.907803:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573394.907812:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573394.907821:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569708544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573394.907828:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573394.907831:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573394.907832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573394.907835:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10148 00000100:00080000:0.0:1788573394.907836:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573394.907839:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573394.907841:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573394.907851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569708672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573394.907853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569708416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573394.907861:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573394.907865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573394.907866:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573394.907868:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10148 00000100:00080000:0.0:1788573394.907870:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573394.907872:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573394.907875:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573394.907885:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569708800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573394.907896:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573394.907899:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573394.907900:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573394.907903:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10148 00000100:00080000:0.0:1788573394.907905:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573394.907907:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573394.907910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573394.907916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569708928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573394.907944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573394.907989:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573394/real 1788573394] req@ffff920a48350700 x1875444569708416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573394.908018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569708416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573394.908031:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573394.908036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573394.908040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569708416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573394.908049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573394.908074:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573394.908089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573394/real 1788573394] req@ffff920a48350380 x1875444569708544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573394.908103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569708544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573394.908113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573394.908117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573394.908120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573394.908149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573394.908172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573394.908184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573394/real 1788573394] req@ffff920a48353480 x1875444569708672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573394.908197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569708672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573394.908213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573394.908217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573394.908221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573394.908227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573394.908250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573394.908262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573394/real 1788573394] req@ffff920a48353800 x1875444569708800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573394.908274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569708800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573394.908284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573394.908287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573394.908291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573394.908298:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573394.908319:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573394.908332:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573394/real 1788573394] req@ffff920a48353b80 x1875444569708928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573394.908344:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569708928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573394.908353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573394.908357:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573394.908361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569708928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573395.355760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573395.931716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573396.379810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573396.955724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573397.403810:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573397.979748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573398.427715:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573399.003762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573399.451797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573400.027724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573400.027728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573400.027739:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573400.027744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573400.027755:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10153 00000100:00080000:0.0:1788573400.027767:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573400.027775:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573400.027822:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569709056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573400.027850:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573400.027854:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573400.027856:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573400.027860:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10153 00000100:00080000:0.0:1788573400.027865:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573400.027871:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573400.027884:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569709056:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573400.027885:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573400.027905:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569709184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573400.027917:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573400.027921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573400.027923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573400.027928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10153 00000100:00080000:0.0:1788573400.027933:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573400.027938:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573400.027944:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573400.027958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569709312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573400.027964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573400.027973:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573400.027976:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573400.027978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573400.027982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10153 00000100:00080000:0.0:1788573400.027986:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573400.027989:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573400.027994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573400.028003:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569709440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573400.028010:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff920a48351500 x1875444569709056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573400.028012:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573400.028021:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573400.028024:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573400.028031:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10153 00000100:00080000:0.0:1788573400.028033:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573400.028038:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573400.028038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569709056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573400.028046:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573400.028058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573400.028061:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569709568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573400.028065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573400.028074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569709056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573400.028100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573400.028118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573400.028132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff920a48351c00 x1875444569709184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573400.028145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569709184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573400.028154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573400.028158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573400.028161:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573400.028168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573400.028189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573400.028202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff920a48350a80 x1875444569709312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573400.028216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569709312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573400.028222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573400.028224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573400.028226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573400.028230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573400.028246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573400.028253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff920a48353b80 x1875444569709440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573400.028260:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569709440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573400.028265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573400.028267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573400.028269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573400.028272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573400.028283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573400.028288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573400/real 1788573400] req@ffff920a48353800 x1875444569709568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573400.028301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569709568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573400.028306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573400.028308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573400.028310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573400.475778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573401.051713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573401.499747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573402.075757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573402.523798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573403.099811:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573403.547816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573404.123735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573404.571866:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573405.147709:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573405.147709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573405.147721:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573405.147726:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573405.147738:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10158 00000100:00080000:0.0:1788573405.147751:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573405.147761:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573405.147809:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569709696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573405.147836:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573405.147839:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573405.147841:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573405.147844:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10158 00000100:00080000:0.0:1788573405.147847:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573405.147850:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573405.147860:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573405.147870:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569709824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573405.147883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573405.147886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573405.147888:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573405.147888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569709696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573405.147895:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10158 00000100:00080000:0.0:1788573405.147899:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573405.147902:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573405.147905:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573405.147912:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569709952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573405.147919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573405.147921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573405.147922:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573405.147925:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10158 00000100:00080000:0.0:1788573405.147927:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573405.147929:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573405.147932:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573405.147941:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569710080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573405.147947:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573405.147949:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573405.147950:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573405.147953:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10158 00000100:00080000:0.0:1788573405.147957:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573405.147959:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573405.147963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573405.147971:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569710208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573405.147980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573405.148028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff920a48353480 x1875444569709696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573405.148050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569709696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573405.148063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573405.148068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573405.148072:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569709696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573405.148100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573405.148120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573405.148130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff920a48350380 x1875444569709824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573405.148154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569709824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573405.148163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573405.148167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573405.148171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573405.148178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573405.148209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573405.148222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff920a48350700 x1875444569709952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573405.148236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569709952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573405.148254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573405.148258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573405.148261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569709952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573405.148269:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573405.148291:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573405.148303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff920a48353800 x1875444569710080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573405.148316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569710080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573405.148331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573405.148335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573405.148338:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573405.148345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573405.148376:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573405.148389:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573405/real 1788573405] req@ffff920a48353b80 x1875444569710208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573405.148402:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569710208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573405.148410:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573405.148412:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573405.148414:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573405.595818:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573406.171762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573406.619827:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573407.195729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573407.643718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573408.219723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573408.667771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573409.243744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573409.691823:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573410.267752:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573410.267752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573410.267762:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573410.267777:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.267788:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10163 00000100:00080000:0.0:1788573410.267801:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.267814:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573410.267853:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569710336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.267877:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.267881:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573410.267882:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.267886:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10163 00000100:00080000:0.0:1788573410.267891:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.267895:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573410.267903:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573410.267911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569710336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573410.267917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569710464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.267929:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.267933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573410.267935:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.267938:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10163 00000100:00080000:0.0:1788573410.267941:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.267943:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573410.267947:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573410.267956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569710592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.267967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.267971:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573410.267976:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.267982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10163 00000100:00080000:0.0:1788573410.267985:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.267992:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573410.268000:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573410.268012:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569710720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573410.268021:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573410.268025:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573410.268030:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573410.268035:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10163 00000100:00080000:0.0:1788573410.268041:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573410.268048:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573410.268049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710464:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788573410.268055:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573410.268069:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569710848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573410.268160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573410.268180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573410.268206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573410.715805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573411.291718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573411.739785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573412.315675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573412.763726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573413.339739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788573413.339970:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788573413.340013:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573413.340029:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573413.340066:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573413.340073:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573413.340079:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573413.340085:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788573413.340128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573413] req@ffff920a48353800 x1875444569710848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573413.340161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569710848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573413.340174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573413.340180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573413.340185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710848:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573413.340194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573413] req@ffff920a48351c00 x1875444569710464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573413.340207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569710464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573413.340215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573413.340219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573413.340223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573413.340230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573413] req@ffff920a48351500 x1875444569710592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573413.340243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569710592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573413.340251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573413.340256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573413.340260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573413.340267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573413] req@ffff920a48353b80 x1875444569710720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573413.340281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569710720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573413.340290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573413.340294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573413.340298:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569710720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573413.340306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573410/real 1788573413] req@ffff920a48350a80 x1875444569710336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573413.340319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569710336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573413.340329:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573413.340333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573413.340336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569710336:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788573413.787777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573414.363686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573414.811769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573415.387700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573415.387759:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573415.387765:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573415.387768:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573415.387777:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10168 00000100:00080000:2.0:1788573415.387787:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573415.387795:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573415.387859:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569710976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573415.387887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573415.387891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573415.387893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573415.387898:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10168 00000100:00080000:2.0:1788573415.387902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573415.387906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573415.387915:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573415.387926:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569711104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573415.387934:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573415.387937:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573415.387939:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573415.387943:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10168 00000100:00080000:2.0:1788573415.387948:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573415.387948:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569710976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573415.387954:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573415.387960:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573415.387968:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569711232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573415.387975:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573415.387978:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573415.387980:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573415.387984:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10168 00000100:00080000:2.0:1788573415.387988:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573415.387992:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573415.387998:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573415.388010:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569711360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573415.388019:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573415.388022:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573415.388024:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573415.388029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573415.388033:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10168 00000100:00080000:2.0:1788573415.388039:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573415.388044:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573415.388049:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573415.388063:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569711488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573415.388077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff920a60ac6a00 x1875444569710976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573415.388098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569710976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573415.388110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573415.388114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573415.388118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569710976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573415.388127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573415.388155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573415.388170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff920a60ac5180 x1875444569711104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573415.388191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569711104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573415.388200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573415.388205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573415.388209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573415.388236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573415.388260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573415.388273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff920a60ac7100 x1875444569711232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573415.388287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569711232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573415.388296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573415.388300:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573415.388303:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573415.388310:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573415.388332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573415.388345:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff920a60ac6300 x1875444569711360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573415.388357:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569711360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573415.388366:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573415.388369:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573415.388373:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573415.388380:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573415.388403:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573415.388416:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573415/real 1788573415] req@ffff920a60ac5500 x1875444569711488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573415.388429:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569711488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573415.388438:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573415.388442:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573415.388446:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573415.835769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573416.411717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573416.859747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573417.435676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573417.883775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573418.459706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573418.907724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573419.483722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573419.931816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573420.507719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573420.507735:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573420.507746:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573420.507751:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573420.507761:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10173 00000100:00080000:2.0:1788573420.507770:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573420.507776:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573420.507808:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569711616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573420.507830:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573420.507833:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573420.507834:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573420.507838:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10173 00000100:00080000:2.0:1788573420.507840:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573420.507842:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573420.507850:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573420.507860:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569711744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573420.507867:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573420.507870:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573420.507872:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573420.507874:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10173 00000100:00080000:2.0:1788573420.507876:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573420.507879:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573420.507882:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573420.507891:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569711872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573420.507898:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569711616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573420.507902:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573420.507915:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573420.507916:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573420.507935:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10173 00000100:00080000:2.0:1788573420.507939:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573420.507942:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573420.507947:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573420.507958:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569712000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573420.507965:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573420.507968:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573420.507969:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573420.507972:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10173 00000100:00080000:2.0:1788573420.507975:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573420.507977:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573420.507980:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573420.507986:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569712128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573420.508023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573420.508070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff920a60ac4a80 x1875444569711616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573420.508092:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569711616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573420.508105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573420.508110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573420.508114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569711616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573420.508123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573420.508148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573420.508162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff920a60ac4700 x1875444569711744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573420.508182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569711744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573420.508192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573420.508196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573420.508199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573420.508228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573420.508250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573420.508261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff920a60ac4380 x1875444569711872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573420.508275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569711872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573420.508283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573420.508287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573420.508290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569711872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573420.508297:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573420.508317:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573420.508328:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff920a60ac4000 x1875444569712000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573420.508340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569712000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573420.508355:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573420.508359:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573420.508362:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573420.508368:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573420.508388:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573420.508399:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573420/real 1788573420] req@ffff920a60ac7800 x1875444569712128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573420.508412:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569712128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573420.508421:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573420.508425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573420.508428:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573420.955793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573421.531728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573421.979822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573422.555693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573423.003739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573423.579708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573424.027787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573424.603737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573425.051748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573425.627756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573425.627760:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573425.627775:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573425.627780:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573425.627794:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10179 00000100:00080000:0.0:1788573425.627806:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573425.627815:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573425.627857:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569712256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573425.627886:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573425.627891:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573425.627893:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573425.627898:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10179 00000100:00080000:0.0:1788573425.627903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573425.627909:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573425.627923:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573425.627936:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569712256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573425.627945:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569712384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573425.627963:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573425.627967:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573425.627970:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573425.627977:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10179 00000100:00080000:0.0:1788573425.627982:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573425.627987:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573425.627995:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573425.628010:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569712512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573425.628037:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573425.628042:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788573425.628046:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573425.628048:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573425.628067:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10179 00000100:00080000:0.0:1788573425.628073:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573425.628079:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573425.628087:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573425.628099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573425/real 1788573425] req@ffff920a48350700 x1875444569712256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573425.628107:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569712640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573425.628145:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573425.628150:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573425.628154:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573425.628161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569712256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573425.628167:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10179 00000100:00080000:0.0:1788573425.628183:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573425.628189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573425.628193:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788573425.628196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573425.628205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569712256:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573425.628207:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573425.628229:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569712768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573425.628237:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573425.628275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573425.628289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573425/real 1788573425] req@ffff920a48350380 x1875444569712384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573425.628297:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569712384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573425.628303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573425.628306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573425.628308:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573425.628312:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573425.628324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573425.628330:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573425/real 1788573425] req@ffff920a48353480 x1875444569712512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573425.628337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569712512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573425.628342:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573425.628344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573425.628346:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573425.628349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573425.628362:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573425.628368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573425/real 1788573425] req@ffff920a48350a80 x1875444569712640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573425.628375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569712640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573425.628380:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573425.628382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573425.628384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573425.628395:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573425.628405:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573425.628411:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573425/real 1788573425] req@ffff920a48353b80 x1875444569712768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573425.628418:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569712768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573425.628423:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573425.628425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573425.628427:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569712768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573426.075777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573426.651727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573427.099757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573427.675752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573428.123785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573428.699647:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573429.147781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573429.723712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573430.171740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573430.747677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573430.747685:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573430.747695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573430.747700:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573430.747711:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10184 00000100:00080000:0.0:1788573430.747722:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573430.747730:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573430.747768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569712896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573430.747802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573430.747807:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573430.747809:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573430.747814:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10184 00000100:00080000:0.0:1788573430.747818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573430.747823:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573430.747833:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573430.747849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569713024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573430.747857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569712896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573430.747882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573430.747887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573430.747889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573430.747895:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10184 00000100:00080000:0.0:1788573430.747900:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573430.747905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573430.747911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573430.747918:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573430.747927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569713152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573430.747939:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573430.747943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573430.747946:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573430.747951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573430/real 1788573430] req@ffff920a48351500 x1875444569712896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573430.747955:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10184 00000100:00080000:0.0:1788573430.747963:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573430.747973:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573430.747981:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573430.747983:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569712896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573430.747992:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573430.747997:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569713280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573430.748008:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573430.748015:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569712896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573430.748016:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573430.748023:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573430.748026:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573430.748031:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10184 00000100:00080000:0.0:1788573430.748036:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573430.748038:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713024:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573430.748044:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573430.748050:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573430.748060:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573430.748068:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569713408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573430.748073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573430/real 1788573430] req@ffff920a48351c00 x1875444569713024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573430.748098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569713024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573430.748105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573430.748120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573430.748122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573430.748127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573430.748144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573430.748157:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573430/real 1788573430] req@ffff920a48353800 x1875444569713152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573430.748166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569713152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573430.748176:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573430.748178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573430.748181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573430.748184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573430.748196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573430.748202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573430/real 1788573430] req@ffff920a48353b80 x1875444569713280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573430.748210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569713280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573430.748215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573430.748217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573430.748219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573430.748230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573430.748240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573430.748247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573430/real 1788573430] req@ffff920a48350a80 x1875444569713408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573430.748254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569713408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573430.748259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573430.748261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573430.748264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573431.195755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573431.771661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573432.219758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573432.795735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573433.243805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573433.819677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573434.267771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573434.843719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573435.291785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573435.867746:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573435.867751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573435.867768:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573435.867772:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573435.867784:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10189 00000100:00080000:2.0:1788573435.867795:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573435.867802:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573435.867839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569713536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573435.867856:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573435.867859:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573435.867861:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573435.867865:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10189 00000100:00080000:2.0:1788573435.867867:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573435.867870:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573435.867878:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573435.867890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569713664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573435.867892:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569713536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573435.867906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573435.867908:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573435.867909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573435.867912:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10189 00000100:00080000:2.0:1788573435.867914:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573435.867916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573435.867919:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573435.867929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569713792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573435.867935:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573435.867938:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573435.867939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573435.867942:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10189 00000100:00080000:2.0:1788573435.867952:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573435.867955:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573435.867959:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573435.867964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573435.867972:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569713920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573435.867980:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573435.867982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573435.867984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573435.867986:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10189 00000100:00080000:2.0:1788573435.867988:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573435.867991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573435.867994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573435.868002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573435/real 1788573435] req@ffff920a60ac5c00 x1875444569713536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573435.868003:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569714048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573435.868045:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569713536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573435.868056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573435.868061:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573435.868065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569713536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573435.868088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573435.868109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573435.868120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573435/real 1788573435] req@ffff920a60ac4e00 x1875444569713664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573435.868133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569713664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573435.868143:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573435.868146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573435.868150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573435.868156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573435.868172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573435.868183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573435/real 1788573435] req@ffff920a60ac7800 x1875444569713792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573435.868196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569713792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573435.868218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573435.868222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573435.868225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573435.868232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573435.868253:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573435.868264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573435/real 1788573435] req@ffff920a60ac4000 x1875444569713920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573435.868276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569713920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573435.868283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573435.868287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573435.868289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569713920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573435.868295:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573435.868314:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573435.868327:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573435/real 1788573435] req@ffff920a60ac4380 x1875444569714048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573435.868340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569714048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573435.868349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573435.868352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573435.868356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573436.315824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573436.891700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573437.339770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573437.915861:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573438.363733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573438.939774:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573439.387853:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573439.963679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573440.411735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573440.987703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573440.987713:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573440.987723:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573440.987727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573440.987741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10194 00000100:00080000:2.0:1788573440.987754:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573440.987763:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573440.987811:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569714176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573440.987835:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573440.987838:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573440.987839:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573440.987843:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10194 00000100:00080000:2.0:1788573440.987847:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573440.987850:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573440.987862:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573440.987877:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569714176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573440.987878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569714304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573440.987901:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573440.987905:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573440.987908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573440.987915:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10194 00000100:00080000:2.0:1788573440.987920:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573440.987927:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573440.987934:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573440.987947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569714432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573440.987957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573440.987958:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573440.987967:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573440.987969:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573440.987974:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10194 00000100:00080000:2.0:1788573440.987980:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573440.987983:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573440.987988:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573440.988000:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569714560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573440.988006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573440/real 1788573440] req@ffff920a60ac4700 x1875444569714176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573440.988025:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573440.988052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569714176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573440.988056:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573440.988063:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573440.988066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573440.988073:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10194 00000100:00080000:3.0:1788573440.988075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573440.988086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569714176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573440.988087:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573440.988101:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573440.988107:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573440.988120:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569714688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573440.988128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573440.988161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573440.988176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573440/real 1788573440] req@ffff920a60ac4a80 x1875444569714304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573440.988185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569714304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573440.988196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573440.988199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573440.988201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573440.988205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573440.988217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573440.988223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573440/real 1788573440] req@ffff920a60ac5500 x1875444569714432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573440.988230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569714432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573440.988235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573440.988237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573440.988239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573440.988243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573440.988254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573440.988260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573440/real 1788573440] req@ffff920a60ac6300 x1875444569714560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573440.988267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569714560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573440.988272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573440.988274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573440.988276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573440.988286:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573440.988316:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573440.988325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573440/real 1788573440] req@ffff920a60ac7100 x1875444569714688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573440.988332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569714688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573465 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573440.988338:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573440.988340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573440.988342:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573441.435795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573442.011677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573442.459770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573443.035711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573443.483766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573444.059683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573444.507738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573445.083735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573445.531798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573446.107679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573446.107712:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.107718:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573446.107721:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.107729:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10199 00000100:00080000:0.0:1788573446.107739:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.107745:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573446.107774:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569714816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573446.107795:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.107798:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573446.107801:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.107804:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10199 00000100:00080000:0.0:1788573446.107807:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.107810:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573446.107819:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573446.107830:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569714944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573446.107835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569714816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573446.107840:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.107846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573446.107847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.107852:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10199 00000100:00080000:0.0:1788573446.107855:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.107858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573446.107863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573446.107872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569715072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573446.107879:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.107882:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573446.107883:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.107886:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10199 00000100:00080000:0.0:1788573446.107889:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573446.107893:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573446.107898:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573446.107906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573446.107911:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569715200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573446.107922:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573446.107925:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573446.107928:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573446.107933:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10199 00000100:00080000:0.0:1788573446.107938:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573446.107943:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff920a48353480 x1875444569714816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573446.107945:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573446.107955:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573446.107969:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569714816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573446.107972:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569715328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573446.108002:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573446.108008:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573446.108013:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569714816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573446.108040:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573446.108066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573446.108079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff920a48350380 x1875444569714944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573446.108093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569714944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573446.108101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573446.108105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573446.108108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569714944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573446.108114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573446.108133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573446.108142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff920a48350700 x1875444569715072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573446.108153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569715072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573446.108161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573446.108164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573446.108177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573446.108184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573446.108210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573446.108222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff920a48350a80 x1875444569715200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573446.108234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569715200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573446.108241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573446.108245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573446.108249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573446.108255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573446.108275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573446.108288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573446/real 1788573446] req@ffff920a48353b80 x1875444569715328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573446.108299:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569715328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573446.108307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573446.108311:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573446.108314:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573446.555717:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573447.131712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573447.579759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573448.155723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573448.603758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573449.179670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573449.627753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573450.203700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573450.651686:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573451.227669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573451.227690:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.227698:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573451.227702:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.227710:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10204 00000100:00080000:2.0:1788573451.227720:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.227726:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573451.227760:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569715456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573451.227783:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.227786:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573451.227787:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.227790:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10204 00000100:00080000:2.0:1788573451.227792:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.227795:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573451.227802:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573451.227811:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569715584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573451.227818:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.227821:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573451.227822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.227825:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10204 00000100:00080000:2.0:1788573451.227826:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.227828:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573451.227832:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573451.227839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569715712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573451.227848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.227851:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573451.227851:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569715456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573451.227854:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.227860:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10204 00000100:00080000:2.0:1788573451.227863:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.227865:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573451.227868:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573451.227878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569715840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573451.227884:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573451.227887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573451.227888:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573451.227890:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10204 00000100:00080000:2.0:1788573451.227892:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573451.227894:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573451.227897:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573451.227903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569715968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573451.227919:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573451.227949:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff920a60ac5180 x1875444569715456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573451.227962:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569715456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573451.227969:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573451.227972:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573451.227976:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569715456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573451.227981:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573451.227996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573451.228004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff920a60ac6a00 x1875444569715584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573451.228012:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569715584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573451.228020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573451.228023:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573451.228026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573451.228050:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573451.228072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573451.228083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff920a60ac7100 x1875444569715712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573451.228091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569715712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573451.228097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573451.228099:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573451.228102:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573451.228106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573451.228132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573451.228143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff920a60ac6300 x1875444569715840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573451.228155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569715840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573451.228163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573451.228167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573451.228170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573451.228177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573451.228198:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573451.228210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573451/real 1788573451] req@ffff920a60ac5500 x1875444569715968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573451.228222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569715968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573451.228230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573451.228233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573451.228237:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569715968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573451.675714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573452.251723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573452.699750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573453.275689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573453.723793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573454.299710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573454.747793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573455.323721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573455.771780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573456.347757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573456.347760:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.347772:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573456.347778:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.347789:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10209 00000100:00080000:2.0:1788573456.347800:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.347810:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573456.347847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569716096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.347867:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.347870:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573456.347871:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.347875:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10209 00000100:00080000:2.0:1788573456.347877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.347880:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573456.347889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573456.347898:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569716224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.347908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573456.347911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569716096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573456.347912:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573456.347918:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.347928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10209 00000100:00080000:2.0:1788573456.347930:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.347932:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573456.347935:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573456.347946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569716352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.347957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.347960:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573456.347961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.347965:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10209 00000100:00080000:2.0:1788573456.347984:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.347987:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573456.347990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573456.347996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573456.348001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569716480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573456.348010:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573456.348012:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573456.348016:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573456.348021:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10209 00000100:00080000:2.0:1788573456.348029:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573456.348034:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573456.348039:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573456.348056:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569716608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573456.348056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff920a60ac4a80 x1875444569716096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573456.348115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569716096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573456.348130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573456.348136:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573456.348139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569716096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573456.348166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573456.348188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573456.348199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff920a60ac4700 x1875444569716224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573456.348208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569716224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573456.348215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573456.348217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573456.348220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573456.348224:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573456.348235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573456.348241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff920a60ac4380 x1875444569716352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573456.348252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569716352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573456.348268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573456.348272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573456.348275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573456.348281:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573456.348302:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573456.348314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff920a60ac4000 x1875444569716480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573456.348325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569716480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573456.348333:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573456.348336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573456.348339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573456.348345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573456.348366:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573456.348378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573456/real 1788573456] req@ffff920a60ac7800 x1875444569716608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573456.348390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569716608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573456.348398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573456.348402:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573456.348406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573456.795811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573457.371764:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573457.819780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573458.395771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573458.843854:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573459.419685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573459.867739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573460.443734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573460.891790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573461.467693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573461.467703:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573461.467709:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573461.467713:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573461.467720:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10214 00000100:00080000:0.0:1788573461.467729:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573461.467736:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573461.467766:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569716736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573461.467786:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573461.467789:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573461.467790:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573461.467793:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10214 00000100:00080000:0.0:1788573461.467796:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573461.467799:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573461.467807:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573461.467819:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569716736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573461.467820:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569716864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573461.467834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573461.467836:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573461.467837:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573461.467841:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10214 00000100:00080000:0.0:1788573461.467844:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573461.467846:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573461.467850:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573461.467859:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569716992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573461.467865:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573461.467868:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573461.467869:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573461.467871:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10214 00000100:00080000:0.0:1788573461.467874:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573461.467876:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573461.467879:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573461.467883:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573461.467889:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569717120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573461.467896:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573461.467899:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573461.467901:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573461.467903:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10214 00000100:00080000:0.0:1788573461.467905:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573461.467908:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573461.467912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573461.467919:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573461] req@ffff920a48353800 x1875444569716736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573461.467921:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569717248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573461.467964:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569716736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573461.467973:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573461.467977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573461.467979:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569716736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573461.467998:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573461.468014:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573461.468023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573461] req@ffff920a48351c00 x1875444569716864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573461.468030:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569716864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573461.468036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573461.468038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573461.468040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573461.468044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573461.468054:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573461.468071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573461] req@ffff920a48351500 x1875444569716992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573461.468080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569716992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573461.468085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573461.468088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573461.468090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569716992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573461.468093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573461.468110:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573461.468117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573461] req@ffff920a48353b80 x1875444569717120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573461.468125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569717120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573461.468130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573461.468132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573461.468133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573461.468137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573461.468147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573461.468153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573461/real 1788573461] req@ffff920a48350a80 x1875444569717248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573461.468161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569717248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573461.468166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573461.468168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573461.468170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573461.915774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573462.491701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573462.939819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573463.515744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573463.963806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573464.539736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573464.987796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573465.563705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573466.011737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573466.587692:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573466.587693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573466.587701:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573466.587706:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573466.587714:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10219 00000100:00080000:0.0:1788573466.587730:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573466.587737:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573466.587766:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569717376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573466.587791:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573466.587794:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573466.587796:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573466.587799:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10219 00000100:00080000:0.0:1788573466.587801:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573466.587804:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573466.587812:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573466.587821:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569717504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573466.587829:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573466.587832:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573466.587833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573466.587836:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10219 00000100:00080000:0.0:1788573466.587839:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573466.587843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569717376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573466.587844:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573466.587851:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573466.587858:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569717632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573466.587866:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573466.587868:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573466.587869:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573466.587871:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10219 00000100:00080000:0.0:1788573466.587874:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573466.587876:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573466.587880:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573466.587888:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569717760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573466.587894:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573466.587897:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573466.587898:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573466.587902:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10219 00000100:00080000:0.0:1788573466.587906:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573466.587906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573466.587913:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573466.587916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573466.587925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569717888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573466.587945:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff920a48350700 x1875444569717376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573466.587959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569717376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573466.587965:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573466.587969:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573466.587974:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569717376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573466.587981:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573466.588006:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573466.588020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff920a48350380 x1875444569717504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573466.588033:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569717504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573466.588042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573466.588045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573466.588047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573466.588067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573466.588083:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573466.588090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff920a48353480 x1875444569717632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573466.588098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569717632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573466.588104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573466.588106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573466.588109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573466.588112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573466.588122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573466.588128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff920a48350a80 x1875444569717760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573466.588135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569717760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573466.588141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573466.588142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573466.588144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573466.588147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573466.588157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573466.588162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573466/real 1788573466] req@ffff920a48353b80 x1875444569717888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573466.588170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569717888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573466.588175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573466.588176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573466.588178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569717888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573467.035725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573467.611641:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573468.059755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573468.635692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573469.083739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573469.659723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573470.107802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573470.683674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573471.131722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573471.707723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573471.707756:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573471.707763:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573471.707768:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573471.707778:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10225 00000100:00080000:3.0:1788573471.707789:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573471.707797:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573471.707840:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569718016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573471.707873:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573471.707876:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573471.707877:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573471.707881:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10225 00000100:00080000:3.0:1788573471.707883:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573471.707886:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573471.707895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573471.707906:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569718144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573471.707920:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573471.707925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788573471.707926:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569718016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573471.707927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573471.707935:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10225 00000100:00080000:3.0:1788573471.707939:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573471.707941:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573471.707944:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573471.707952:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569718272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573471.707959:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573471.707962:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573471.707963:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573471.707965:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10225 00000100:00080000:3.0:1788573471.707968:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573471.707970:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573471.707974:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573471.707981:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569718400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573471.707988:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573471.707990:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573471.707992:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573471.707994:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10225 00000100:00080000:3.0:1788573471.708003:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573471.708006:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573471.708010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573471.708016:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573471.708020:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569718528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573471.708057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573471/real 1788573471] req@ffff920a529b3b80 x1875444569718016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573471.708074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569718016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573471.708100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573471.708104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573471.708107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569718016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573471.708132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573471.708175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573471.708185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573471/real 1788573471] req@ffff920a529b0380 x1875444569718144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573471.708193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569718144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573471.708199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573471.708202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573471.708204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573471.708208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573471.708221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573471.708228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573471/real 1788573471] req@ffff920a529b1180 x1875444569718272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573471.708235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569718272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573471.708240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573471.708242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573471.708244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573471.708247:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573471.708259:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573471.708265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573471/real 1788573471] req@ffff920a529b1500 x1875444569718400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573471.708273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569718400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573471.708278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573471.708279:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573471.708281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573471.708285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573471.708295:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573471.708300:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573471/real 1788573471] req@ffff920a529b0e00 x1875444569718528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573471.708308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569718528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573471.708317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573471.708320:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573471.708323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573472.155781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573472.731689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573473.179768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573473.755701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573474.203797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573474.779750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573475.227753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573475.803698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573476.251785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573476.827702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573476.827860:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573476.827875:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573476.827883:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573476.827893:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10230 00000100:00080000:0.0:1788573476.827905:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573476.827913:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573476.827958:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569718656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573476.827986:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573476.827989:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573476.827990:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573476.827993:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10230 00000100:00080000:0.0:1788573476.827996:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573476.828000:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573476.828010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573476.828024:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569718784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573476.828029:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569718656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573476.828053:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573476.828056:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573476.828058:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573476.828071:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10230 00000100:00080000:0.0:1788573476.828074:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573476.828078:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573476.828083:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573476.828094:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569718912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573476.828107:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573476.828110:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573476.828112:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573476.828114:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10230 00000100:00080000:0.0:1788573476.828116:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573476.828118:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573476.828122:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573476.828133:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569719040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573476.828143:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573476.828149:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573476.828152:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573476.828160:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10230 00000100:00100000:2.0:1788573476.828160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718784:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573476.828169:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573476.828177:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573476.828184:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573476.828202:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569719168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573476.828204:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573476.828240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573476.828260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573477.275725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573477.851721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573478.299717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573478.875725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573479.323745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573479.899736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788573479.899909:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788573479.899955:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573479.899973:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573479.900016:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573479.900050:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788573479.900059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573476/real 1788573479] req@ffff920a48351c00 x1875444569718784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788573479.900076:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788573479.900110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569718784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788573479.900116:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00080000:2.0:1788573479.900146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573479.900155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573479.900161:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573479.900173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573476/real 1788573479] req@ffff920a48353800 x1875444569718912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573479.900189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569718912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573479.900198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573479.900204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573479.900207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569718912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573479.900214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573476/real 1788573479] req@ffff920a48353b80 x1875444569719040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573479.900227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569719040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573479.900235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573479.900255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573479.900259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573479.900267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573476/real 1788573479] req@ffff920a48351500 x1875444569718656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573479.900281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569718656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573479.900289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573479.900292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573479.900296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569718656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573479.900329:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573476/real 1788573479] req@ffff920a48350a80 x1875444569719168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573479.900341:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569719168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573479.900349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573479.900353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573479.900356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573480.347807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573480.923756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573481.371742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573481.947729:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573481.947741:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573481.947745:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573481.947758:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10235 00000100:00080000:1.0:1788573481.947770:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573481.947779:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573481.947801:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788573481.947820:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569719296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573481.947855:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573481.947859:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573481.947860:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573481.947865:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10235 00000100:00080000:1.0:1788573481.947869:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573481.947873:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573481.947884:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573481.947894:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569719296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573481.947901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569719424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573481.947915:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573481.947919:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573481.947921:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573481.947926:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10235 00000100:00080000:1.0:1788573481.947931:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573481.947936:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573481.947941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573481.947957:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569719552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573481.947976:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573481.947980:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573481.947983:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573481.947988:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10235 00000100:00080000:1.0:1788573481.947993:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573481.947998:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573481.948002:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573481.948010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573481.948025:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569719680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573481.948033:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573481.948036:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573481.948037:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573481.948043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573481/real 1788573481] req@ffff920a52893480 x1875444569719296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573481.948043:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10235 00000100:00080000:1.0:1788573481.948049:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573481.948056:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573481.948066:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573481.948080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569719296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573481.948082:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569719808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573481.948114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573481.948122:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573481.948127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569719296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573481.948166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573481.948195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573481.948211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573481/real 1788573481] req@ffff920a52890a80 x1875444569719424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573481.948225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569719424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573481.948235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573481.948239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573481.948242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573481.948250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573481.948273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573481.948286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573481/real 1788573481] req@ffff920a52891f80 x1875444569719552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573481.948299:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569719552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573481.948306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573481.948310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573481.948313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573481.948320:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573481.948343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573481.948355:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573481/real 1788573481] req@ffff920a52892a00 x1875444569719680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573481.948367:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569719680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573481.948376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573481.948381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573481.948385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573481.948391:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573481.948420:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573481.948432:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573481/real 1788573481] req@ffff920a52893800 x1875444569719808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573481.948452:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569719808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573506 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573481.948461:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573481.948464:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573481.948467:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569719808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573482.395814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573482.971846:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573483.420822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573483.995714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573484.443786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573485.019694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573485.467790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573486.043656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573486.491723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573487.067694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573487.067723:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.067729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573487.067733:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.067741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10240 00000100:00080000:0.0:1788573487.067751:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.067757:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573487.067789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569719936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.067813:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.067815:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573487.067817:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.067820:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10240 00000100:00080000:0.0:1788573487.067822:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.067825:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573487.067834:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573487.067844:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569719936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573487.067847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569720064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.067858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.067860:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573487.067861:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.067865:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10240 00000100:00080000:0.0:1788573487.067867:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.067869:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573487.067873:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573487.067881:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569720192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.067888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.067890:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573487.067891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.067894:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10240 00000100:00080000:0.0:1788573487.067897:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.067899:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573487.067902:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573487.067906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573487.067911:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569720320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573487.067919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573487.067921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573487.067923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573487.067926:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10240 00000100:00080000:0.0:1788573487.067929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573487.067931:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573487.067935:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573487.067941:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff920a48353480 x1875444569719936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573487.067943:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569720448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573487.067985:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569719936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573487.067998:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573487.068003:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573487.068007:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569719936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573487.068032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573487.068052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573487.068062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff920a48350380 x1875444569720064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573487.068070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569720064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573487.068077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573487.068079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573487.068081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573487.068085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573487.068096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573487.068102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff920a48350700 x1875444569720192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573487.068109:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569720192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573487.068118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573487.068120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573487.068122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573487.068125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573487.068136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573487.068141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff920a48350a80 x1875444569720320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573487.068148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569720320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573487.068152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573487.068154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573487.068156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573487.068159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573487.068181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573487.068192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573487/real 1788573487] req@ffff920a48351500 x1875444569720448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573487.068199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569720448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573487.068204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573487.068206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573487.068209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573487.515851:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573488.091703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573488.539778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573489.115672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573489.563765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573490.139707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573490.587764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573491.163704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573491.611790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573492.187691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573492.187713:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.187718:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573492.187722:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.187731:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10245 00000100:00080000:0.0:1788573492.187741:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.187749:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573492.187788:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569720576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.187811:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.187814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573492.187816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.187820:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10245 00000100:00080000:0.0:1788573492.187823:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.187828:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573492.187836:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573492.187839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569720576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573492.187850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569720704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.187861:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.187865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573492.187867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.187872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10245 00000100:00080000:0.0:1788573492.187877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.187881:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573492.187887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573492.187894:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573492.187903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569720832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573492.187912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.187915:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573492.187917:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573492.187921:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10245 00000100:00080000:0.0:1788573492.187925:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573492.187927:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff920a48353b80 x1875444569720576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573492.187930:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573492.187938:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573492.187947:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569720576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573492.187950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569720960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573492.187972:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573492.187973:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573492.187979:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573492.187982:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573492.187982:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788573492.187989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10245 00000100:00100000:2.0:1788573492.187989:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569720576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573492.187994:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573492.187998:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573492.188001:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573492.188009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569721088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573492.188014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573492.188039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573492.188050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff920a48353800 x1875444569720704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.188059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569720704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.188064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.188067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573492.188070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573492.188073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573492.188088:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573492.188099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff920a48351c00 x1875444569720832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.188113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569720832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.188122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.188125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573492.188129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573492.188142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573492.188161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573492.188169:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff920a48351500 x1875444569720960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.188177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569720960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.188182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.188185:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573492.188187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569720960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573492.188198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573492.188209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573492.188215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573492/real 1788573492] req@ffff920a48350a80 x1875444569721088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573492.188222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569721088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573492.188228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573492.188229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573492.188231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573492.635755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573493.211707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573493.659781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573494.235698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573494.683821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573495.259668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573495.707778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573496.283700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573496.731731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573497.307700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573497.307706:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.307713:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573497.307716:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.307724:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10250 00000100:00080000:3.0:1788573497.307734:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.307741:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573497.307775:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569721216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.307791:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.307794:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573497.307796:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.307799:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10250 00000100:00080000:3.0:1788573497.307802:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.307805:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573497.307813:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573497.307819:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569721216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573497.307825:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569721344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.307835:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.307838:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573497.307839:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.307842:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10250 00000100:00080000:3.0:1788573497.307844:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.307846:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573497.307849:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573497.307858:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569721472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.307864:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.307867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573497.307868:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.307870:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10250 00000100:00080000:3.0:1788573497.307880:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.307883:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573497.307886:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573497.307893:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573497.307896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569721600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573497.307904:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573497.307907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573497.307908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573497.307911:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10250 00000100:00080000:3.0:1788573497.307917:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573497.307919:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573497.307922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573497.307930:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569721728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573497.307935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff920a529b1f80 x1875444569721216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.307960:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569721216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.307969:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.307973:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573497.307977:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569721216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573497.307999:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573497.308018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573497.308026:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff920a529b0000 x1875444569721344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.308034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569721344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.308040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.308041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573497.308043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573497.308047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573497.308057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573497.308063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff920a529b3100 x1875444569721472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.308070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569721472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.308075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.308077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573497.308079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573497.308082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573497.308092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573497.308097:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff920a529b2d80 x1875444569721600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.308115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569721600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.308121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.308123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573497.308125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573497.308129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573497.308141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573497.308147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573497/real 1788573497] req@ffff920a529b0700 x1875444569721728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573497.308155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569721728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573497.308160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573497.308161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573497.308163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573497.755703:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573498.331686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573498.779769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573499.355710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573499.803790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573500.379683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573500.827724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573501.403691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573501.851778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573502.427681:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573502.427689:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573502.427693:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573502.427702:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10255 00000100:00100000:1.0:1788573502.427708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573502.427712:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573502.427720:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573502.427754:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569721856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573502.427772:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573502.427775:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573502.427777:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573502.427779:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10255 00000100:00080000:3.0:1788573502.427782:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573502.427784:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573502.427791:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573502.427802:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569721984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573502.427809:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573502.427811:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573502.427813:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573502.427815:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10255 00000100:00080000:3.0:1788573502.427818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573502.427821:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573502.427824:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573502.427832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569721856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573502.427834:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569722112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573502.427849:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573502.427852:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573502.427853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573502.427855:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10255 00000100:00080000:3.0:1788573502.427857:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573502.427859:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573502.427863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573502.427870:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569722240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573502.427876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573502.427879:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573502.427880:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573502.427883:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10255 00000100:00080000:3.0:1788573502.427898:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573502.427900:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573502.427903:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573502.427910:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573502.427912:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569722368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573502.427955:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff920a529b1c00 x1875444569721856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.427977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569721856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.427990:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.427994:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573502.427999:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569721856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573502.428007:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573502.428034:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573502.428050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff920a529b2300 x1875444569721984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.428065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569721984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.428075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.428079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573502.428082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569721984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573502.428110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573502.428134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573502.428147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff920a529b3480 x1875444569722112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.428161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569722112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.428171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.428175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573502.428178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573502.428185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573502.428208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573502.428221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff920a529b2a00 x1875444569722240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.428245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569722240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.428255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.428259:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573502.428262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573502.428270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573502.428292:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573502.428306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573502/real 1788573502] req@ffff920a529b0700 x1875444569722368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573502.428319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569722368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573502.428328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573502.428331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573502.428335:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573502.875711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573503.451705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573503.899821:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573504.475712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573504.923758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573505.499723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573505.947719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573506.523673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573506.971792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573507.547673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573507.547673:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.547682:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573507.547686:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.547695:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10260 00000100:00080000:2.0:1788573507.547706:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.547714:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573507.547748:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569722496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573507.547771:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.547774:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573507.547776:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.547779:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10260 00000100:00080000:2.0:1788573507.547782:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.547785:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573507.547794:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573507.547805:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569722496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573507.547806:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569722624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573507.547821:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.547825:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573507.547827:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.547832:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10260 00000100:00080000:2.0:1788573507.547837:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.547842:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573507.547848:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573507.547863:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569722752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573507.547871:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573507.547880:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573507.547886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573507.547888:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.547894:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10260 00000100:00080000:2.0:1788573507.547899:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.547904:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573507.547910:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff920a60ac4e00 x1875444569722496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788573507.547912:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573507.547933:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569722880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573507.547934:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569722496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573507.547963:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573507.547965:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573507.547969:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788573507.547974:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788573507.547975:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569722496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573507.547978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573507.547985:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10260 00000100:00080000:2.0:1788573507.547989:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573507.547994:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573507.547999:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722624:192.168.202.157@tcp:38:kworker.0 00000008:00100000:2.0:1788573507.548000:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573507.548019:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569723008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573507.548026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573507.548041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff920a60ac5c00 x1875444569722624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573507.548051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569722624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573507.548058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573507.548060:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573507.548063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573507.548067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573507.548077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573507.548083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff920a60ac7800 x1875444569722752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573507.548091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569722752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573507.548096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573507.548098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573507.548099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573507.548102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573507.548113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573507.548118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff920a60ac4000 x1875444569722880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573507.548125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569722880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573507.548131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573507.548133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573507.548135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569722880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573507.548144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573507.548154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573507.548160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573507/real 1788573507] req@ffff920a60ac4380 x1875444569723008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573507.548166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569723008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573507.548172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573507.548173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573507.548175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573507.995731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573508.571766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573509.019782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573509.595664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573510.043711:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573510.619748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573511.067733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573511.643688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573512.091782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573512.667689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573512.667728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573512.667733:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573512.667736:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573512.667745:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10266 00000100:00080000:0.0:1788573512.667752:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573512.667758:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573512.667789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569723136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573512.667820:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573512.667824:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573512.667827:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573512.667832:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10266 00000100:00080000:0.0:1788573512.667836:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573512.667842:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573512.667853:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573512.667869:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569723264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573512.667884:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573512.667889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788573512.667890:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569723136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573512.667891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573512.667902:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10266 00000100:00080000:0.0:1788573512.667912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573512.667916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573512.667921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573512.667933:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569723392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573512.667940:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573512.667942:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573512.667944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573512.667947:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10266 00000100:00080000:0.0:1788573512.667958:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573512.667961:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573512.667965:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573512.667973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569723520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573512.667981:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573512.667984:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573512.667985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573512.667987:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10266 00000100:00080000:0.0:1788573512.667989:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573512.667991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573512.667995:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573512.668002:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573512.668015:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569723648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573512.668064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573512/real 1788573512] req@ffff920a48350700 x1875444569723136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573512.668089:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569723136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573512.668102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573512.668107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573512.668111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569723136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573512.668145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573512.668173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573512.668188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573512/real 1788573512] req@ffff920a48350380 x1875444569723264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573512.668202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569723264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573512.668212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573512.668216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573512.668219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573512.668227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573512.668247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573512.668259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573512/real 1788573512] req@ffff920a48353480 x1875444569723392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573512.668279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569723392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573512.668288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573512.668291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573512.668294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573512.668301:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573512.668321:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573512.668332:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573512/real 1788573512] req@ffff920a48350a80 x1875444569723520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573512.668344:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569723520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573512.668353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573512.668356:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573512.668360:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573512.668366:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573512.668385:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573512.668396:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573512/real 1788573512] req@ffff920a48351500 x1875444569723648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573512.668409:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569723648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573512.668425:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573512.668429:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573512.668433:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573513.115798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573513.691737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573514.139756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573514.715738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573515.163768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573515.739718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573516.187778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573516.763745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573517.211792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573517.787711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573517.787750:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573517.787757:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573517.787761:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573517.787772:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10271 00000100:00080000:3.0:1788573517.787783:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573517.787791:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573517.787833:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569723776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573517.787859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573517.787862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573517.787864:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573517.787867:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10271 00000100:00080000:3.0:1788573517.787870:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573517.787873:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573517.787884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573517.787895:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569723904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573517.787904:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573517.787906:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573517.787908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573517.787913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569723776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573517.787915:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10271 00000100:00080000:3.0:1788573517.787922:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573517.787926:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573517.787931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573517.787942:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569724032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573517.787949:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573517.787952:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573517.787954:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573517.787958:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10271 00000100:00080000:3.0:1788573517.787961:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573517.787964:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573517.787969:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573517.787977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569724160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573517.787984:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573517.787988:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573517.787989:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573517.787992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10271 00000100:00080000:3.0:1788573517.787996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788573517.788001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573517.788003:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573517.788007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573517.788016:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569724288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573517.788041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573517/real 1788573517] req@ffff920a529b2d80 x1875444569723776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573517.788056:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569723776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573517.788066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573517.788069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573517.788073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569723776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573517.788098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573517.788120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573517.788134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573517/real 1788573517] req@ffff920a529b3100 x1875444569723904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573517.788147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569723904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573517.788157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573517.788161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573517.788165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569723904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573517.788171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573517.788188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573517.788196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573517/real 1788573517] req@ffff920a529b0000 x1875444569724032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573517.788204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569724032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573517.788210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573517.788211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573517.788213:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573517.788217:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573517.788230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573517.788236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573517/real 1788573517] req@ffff920a529b1f80 x1875444569724160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573517.788245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569724160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573517.788253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573517.788256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573517.788259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573517.788266:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573517.788288:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573517.788301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573517/real 1788573517] req@ffff920a529b0e00 x1875444569724288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573517.788310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569724288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573517.788316:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573517.788318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573517.788320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573518.235749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573518.811736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573519.259746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573519.835681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573520.283740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573520.859708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573521.307731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573521.883736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573522.331711:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573522.907696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573522.907697:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573522.907706:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573522.907710:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573522.907718:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10276 00000100:00080000:0.0:1788573522.907727:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573522.907734:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573522.907768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569724416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573522.907795:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573522.907798:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573522.907799:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573522.907802:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10276 00000100:00080000:0.0:1788573522.907805:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573522.907807:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573522.907816:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573522.907826:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569724416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573522.907828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569724544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573522.907839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573522.907842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573522.907843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573522.907846:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10276 00000100:00080000:0.0:1788573522.907849:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573522.907851:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573522.907854:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573522.907864:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569724672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573522.907874:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573522.907877:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573522.907878:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573522.907880:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10276 00000100:00080000:0.0:1788573522.907883:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573522.907885:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573522.907889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573522.907889:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573522.907900:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569724800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573522.907908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573522.907910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573522.907911:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573522.907915:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10276 00000100:00080000:0.0:1788573522.907918:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573522.907922:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573522.907927:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573522.907927:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573522/real 1788573522] req@ffff920a48351c00 x1875444569724416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573522.907939:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569724928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573522.907947:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569724416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573522.907956:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573522.907960:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573522.907963:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569724416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573522.907984:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573522.908000:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573522.908026:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573522/real 1788573522] req@ffff920a48353800 x1875444569724544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573522.908041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569724544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573522.908051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573522.908056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573522.908060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573522.908067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573522.908093:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573522.908105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573522/real 1788573522] req@ffff920a48353b80 x1875444569724672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573522.908115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569724672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573522.908121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573522.908124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573522.908126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573522.908131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573522.908156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573522.908163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573522/real 1788573522] req@ffff920a48351500 x1875444569724800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573522.908188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569724800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573522.908195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573522.908197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573522.908199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573522.908203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573522.908215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573522.908222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573522/real 1788573522] req@ffff920a48350a80 x1875444569724928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573522.908230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569724928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573522.908251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573522.908253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573522.908254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569724928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573523.355763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573523.931658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573524.379792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573524.955708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573525.403816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573525.979718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573526.427787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573527.003781:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573527.451759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573528.027740:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573528.027740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573528.027752:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573528.027756:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573528.027767:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10281 00000100:00080000:1.0:1788573528.027778:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573528.027786:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573528.027832:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569725056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573528.027858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573528.027861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573528.027862:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573528.027866:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10281 00000100:00080000:1.0:1788573528.027868:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573528.027872:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573528.027882:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573528.027891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569725056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573528.027896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569725184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573528.027910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573528.027912:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573528.027914:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573528.027916:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10281 00000100:00080000:1.0:1788573528.027919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573528.027921:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573528.027925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573528.027934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569725312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573528.027941:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573528.027944:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573528.027945:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573528.027947:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10281 00000100:00080000:1.0:1788573528.027950:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573528.027952:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573528.027955:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573528.027963:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569725440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573528.027969:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573528.027970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573528.027975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573528.027977:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573528.027980:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10281 00000100:00080000:1.0:1788573528.027983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573528.027987:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573528.027993:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573528.028006:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569725568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573528.028024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573528] req@ffff920a52890e00 x1875444569725056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573528.028042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569725056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573528.028051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573528.028054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573528.028058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569725056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573528.028082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573528.028098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573528.028107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573528] req@ffff920a52891180 x1875444569725184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573528.028120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569725184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573528.028129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573528.028133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573528.028136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573528.028142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573528.028164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573528.028177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573528] req@ffff920a52890700 x1875444569725312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573528.028187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569725312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573528.028197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573528.028200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573528.028202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573528.028206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573528.028222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573528.028228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573528] req@ffff920a52893100 x1875444569725440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573528.028236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569725440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573528.028241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573528.028243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573528.028245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573528.028248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573528.028259:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573528.028265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573528/real 1788573528] req@ffff920a52891880 x1875444569725568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573528.028272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569725568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573528.028277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573528.028278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573528.028281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573528.475736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573529.051701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573529.499693:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573530.075750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573530.523795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573531.099703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573531.547768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573532.123780:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573532.571783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573533.147693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573533.147694:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573533.147706:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573533.147711:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573533.147724:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10286 00000100:00080000:1.0:1788573533.147736:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573533.147744:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573533.147805:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569725696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573533.147834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573533.147838:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573533.147839:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573533.147842:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10286 00000100:00080000:1.0:1788573533.147846:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573533.147850:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573533.147860:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573533.147872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569725696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573533.147874:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569725824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573533.147894:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573533.147898:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573533.147899:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573533.147906:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10286 00000100:00080000:1.0:1788573533.147908:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573533.147912:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573533.147916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573533.147931:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569725952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573533.147938:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573533.147940:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573533.147942:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573533.147946:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10286 00000100:00080000:1.0:1788573533.147949:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573533.147955:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573533.147961:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573533.147967:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573533.147975:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569726080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573533.147983:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573533.147986:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573533.147987:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573533.147999:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10286 00000100:00100000:2.0:1788573533.148003:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff920a52892d80 x1875444569725696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573533.148006:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573533.148019:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573533.148035:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573533.148044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569725696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573533.148050:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569726208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573533.148087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573533.148096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573533.148100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569725696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573533.148132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573533.148151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573533.148161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff920a52892300 x1875444569725824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573533.148179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569725824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573533.148185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573533.148188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573533.148190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573533.148195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573533.148206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573533.148212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff920a52891880 x1875444569725952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573533.148220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569725952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573533.148225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573533.148227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573533.148229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569725952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573533.148233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573533.148252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573533.148259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff920a52893100 x1875444569726080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573533.148266:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569726080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573533.148271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573533.148273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573533.148275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573533.148279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573533.148290:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573533.148296:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573533/real 1788573533] req@ffff920a52890700 x1875444569726208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573533.148304:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569726208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573533.148308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573533.148310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573533.148312:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573533.595830:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573534.171728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573534.619787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573535.195690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573535.643765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573536.219691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573536.667765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573537.243685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573537.691760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573538.267746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573538.267789:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.267798:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573538.267802:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.267813:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10291 00000100:00080000:1.0:1788573538.267824:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.267832:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573538.267874:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569726336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.267897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.267900:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573538.267901:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.267905:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10291 00000100:00080000:1.0:1788573538.267907:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.267910:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573538.267920:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573538.267931:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569726464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573538.267939:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569726336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573538.267953:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.267958:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573538.267959:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.267964:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10291 00000100:00080000:1.0:1788573538.267967:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.267971:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573538.267974:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573538.267984:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569726592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573538.267991:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.267994:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573538.267995:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.268000:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10291 00000100:00080000:1.0:1788573538.268004:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.268006:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573538.268010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573538.268018:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569726720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573538.268029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573538.268039:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573538.268045:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573538.268047:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573538.268053:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10291 00000100:00080000:1.0:1788573538.268058:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573538.268063:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573538.268070:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573538.268078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff920a52891180 x1875444569726336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573538.268090:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569726848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573538.268143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569726336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.268158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.268163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573538.268169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569726336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573538.268192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573538.268212:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573538.268222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff920a52890e00 x1875444569726464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.268231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569726464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.268236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.268239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573538.268242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573538.268246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573538.268257:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573538.268263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff920a52893800 x1875444569726592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.268270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569726592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.268276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.268278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573538.268280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573538.268284:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573538.268295:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573538.268301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff920a52892a00 x1875444569726720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.268308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569726720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.268313:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.268315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573538.268317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573538.268321:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573538.268330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573538.268336:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573538/real 1788573538] req@ffff920a52891f80 x1875444569726848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573538.268342:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569726848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573538.268347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573538.268349:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573538.268351:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569726848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573538.715764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573539.291705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573539.739721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573540.315703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573540.763731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573541.339703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573541.787754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573542.363708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573542.811881:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573543.387727:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.387736:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573543.387740:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.387750:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10296 00000100:00080000:1.0:1788573543.387759:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.387766:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573543.387784:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788573543.387800:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569726976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.387831:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.387836:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573543.387839:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.387845:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10296 00000100:00080000:1.0:1788573543.387849:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.387854:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573543.387860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569726976:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573543.387866:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573543.387882:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569727104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.387891:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.387894:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573543.387896:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.387899:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10296 00000100:00080000:1.0:1788573543.387902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.387905:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573543.387908:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573543.387917:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569727232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.387924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.387926:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573543.387927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.387931:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10296 00000100:00080000:1.0:1788573543.387933:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.387935:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573543.387939:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573543.387946:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569727360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573543.387952:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573543.387954:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573543.387956:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573543.387962:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10296 00000100:00100000:2.0:1788573543.387964:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727104:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573543.387965:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573543.387971:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573543.387976:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573543.387986:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569727488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573543.388008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573543.388032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573543.388053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573543.835833:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573544.411712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573544.859768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573545.435711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573545.883827:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573546.459683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573546.461211:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573546.461258:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573546.461286:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573546.461327:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573546.461336:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573546.461345:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573546.461354:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788573546.461447:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573546] req@ffff920a52890380 x1875444569727488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573546.461474:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569727488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573546.461487:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573546.461492:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573546.461496:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727488:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573546.461503:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573546] req@ffff920a52893480 x1875444569727104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573546.461524:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569727104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573546.461534:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573546.461539:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573546.461543:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573546.461559:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573546] req@ffff920a52891500 x1875444569727232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573546.461577:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569727232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573546.461665:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573546.461670:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573546.461674:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573546.461682:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573546] req@ffff920a52892680 x1875444569727360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573546.461696:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569727360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573546.461702:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573546.461705:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573546.461707:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573546.461711:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573543/real 1788573546] req@ffff920a52890a80 x1875444569726976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573546.461722:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569726976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573546.461729:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573546.461738:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573546.461744:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569726976:192.168.202.157@tcp:250:kworker.0 00010000:00010000:0.0:1788573546.907729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573547.483671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573547.931811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573548.507661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573548.507685:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573548.507694:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573548.507699:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573548.507709:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10301 00000100:00080000:1.0:1788573548.507719:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573548.507727:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573548.507761:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569727616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573548.507787:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573548.507790:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573548.507792:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573548.507796:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10301 00000100:00080000:1.0:1788573548.507800:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573548.507804:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573548.507813:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573548.507814:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569727616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573548.507825:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569727744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573548.507834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573548.507837:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573548.507839:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573548.507842:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10301 00000100:00080000:1.0:1788573548.507845:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573548.507849:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573548.507855:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573548.507867:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569727872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573548.507879:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573548.507882:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573548.507884:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788573548.507890:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573548.507891:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10301 00000100:00080000:1.0:1788573548.507897:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573548.507903:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573548.507910:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573548.507924:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569728000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573548.507934:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573548.507935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff920a52893b80 x1875444569727616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573548.507939:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573548.507943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573548.507953:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10301 00000100:00080000:1.0:1788573548.507958:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573548.507963:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573548.507968:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573548.507971:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569727616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573548.507983:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569728128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573548.507985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573548.507995:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573548.508003:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569727616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573548.508033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573548.508059:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573548.508074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff920a52890000 x1875444569727744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573548.508101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569727744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.508110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573548.508114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573548.508117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573548.508124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573548.508148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573548.508160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff920a52890a80 x1875444569727872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573548.508173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569727872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.508188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573548.508191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573548.508195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569727872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573548.508208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573548.508231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573548.508243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff920a52892680 x1875444569728000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573548.508255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569728000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.508263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573548.508266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573548.508269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573548.508275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573548.508297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573548.508309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573548/real 1788573548] req@ffff920a52891500 x1875444569728128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573548.508321:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569728128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573548.508329:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573548.508332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573548.508337:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573548.955790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573549.531703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573549.979791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573550.555648:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573551.003797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573551.579680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573552.027792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573552.603734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573553.051788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573553.627762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573553.627764:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573553.627772:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573553.627777:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573553.627786:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10307 00000100:00080000:3.0:1788573553.627797:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573553.627803:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573553.627842:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569728256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573553.627873:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573553.627878:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573553.627881:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573553.627886:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10307 00000100:00080000:3.0:1788573553.627891:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573553.627896:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573553.627908:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573553.627925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569728384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573553.627937:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573553.627941:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573553.627943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573553.627948:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10307 00000100:00080000:3.0:1788573553.627952:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573553.627957:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573553.627963:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573553.627970:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569728256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573553.627977:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569728512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573553.627995:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573553.627999:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573553.628001:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573553.628006:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10307 00000100:00080000:3.0:1788573553.628011:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573553.628017:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573553.628030:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573553.628058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573553/real 1788573553] req@ffff920a529b1500 x1875444569728256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573553.628072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569728256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573553.628082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573553.628088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573553.628093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569728256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573553.628099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728384:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788573553.628106:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573553.628129:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569728640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573553.628133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573553.628152:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573553.628153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573553/real 1788573553] req@ffff920a529b1180 x1875444569728384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573553.628163:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573553.628182:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573553.628191:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10307 00000100:00100000:1.0:1788573553.628196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569728384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573553.628198:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573553.628211:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573553.628216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573553.628222:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573553.628225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573553.628232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573553.628243:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569728768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573553.628282:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573553.628311:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573553.628324:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573553/real 1788573553] req@ffff920a529b0380 x1875444569728512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573553.628339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569728512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573553.628348:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573553.628352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573553.628356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573553.628363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573553.628386:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573553.628399:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573553/real 1788573553] req@ffff920a529b3b80 x1875444569728640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573553.628414:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569728640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573553.628424:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573553.628428:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573553.628433:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573553.628440:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573553.628461:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573553.628474:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573553/real 1788573553] req@ffff920a529b0e00 x1875444569728768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573553.628489:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569728768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573578 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573553.628499:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573553.628503:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573553.628508:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569728768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573554.075783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573554.651689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573555.099760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573555.675701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573556.123773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573556.699714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573557.147755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573557.723727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573558.171834:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573558.747707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573558.747707:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573558.747720:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573558.747726:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573558.747736:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10312 00000100:00080000:3.0:1788573558.747749:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573558.747757:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573558.747800:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569728896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573558.747828:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573558.747831:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573558.747833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573558.747838:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10312 00000100:00080000:3.0:1788573558.747843:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573558.747849:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573558.747863:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573558.747873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569728896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573558.747900:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569729024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573558.747911:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573558.747914:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573558.747915:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573558.747920:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10312 00000100:00080000:3.0:1788573558.747924:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573558.747927:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573558.747932:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573558.747942:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569729152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573558.747948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573558.747960:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573558.747964:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573558.747966:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573558.747971:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10312 00000100:00080000:3.0:1788573558.747975:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573558.747978:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573558.747983:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573558.747996:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569729280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573558.747998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573558/real 1788573558] req@ffff920a529b1f80 x1875444569728896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573558.748021:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573558.748038:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573558.748046:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573558.748056:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10312 00000100:00080000:3.0:1788573558.748059:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573558.748063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569728896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573558.748073:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573558.748075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573558.748080:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573558.748083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573558.748093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569728896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573558.748099:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569729408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573558.748126:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573558.748152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573558.748162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573558/real 1788573558] req@ffff920a529b0000 x1875444569729024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573558.748175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569729024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573558.748181:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573558.748184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573558.748186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573558.748190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573558.748203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573558.748209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573558/real 1788573558] req@ffff920a529b3100 x1875444569729152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573558.748217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569729152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573558.748222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573558.748224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573558.748226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573558.748230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573558.748241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573558.748246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573558/real 1788573558] req@ffff920a529b2d80 x1875444569729280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573558.748253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569729280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573558.748263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573558.748265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573558.748268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573558.748271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573558.748283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573558.748288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573558/real 1788573558] req@ffff920a529b0700 x1875444569729408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573558.748295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569729408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573558.748300:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573558.748301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573558.748303:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573559.195759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573559.771775:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573560.219757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573560.795679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573561.243730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573561.819755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573562.267776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573562.843681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573563.291827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573563.867718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573563.867744:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573563.867753:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573563.867759:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573563.867772:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10317 00000100:00080000:0.0:1788573563.867786:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573563.867796:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573563.867851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569729536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573563.867882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573563.867887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573563.867889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573563.867894:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10317 00000100:00080000:0.0:1788573563.867899:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573563.867905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573563.867917:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573563.867919:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569729536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573563.867936:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569729664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573563.867950:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573563.867954:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573563.867956:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573563.867961:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10317 00000100:00080000:0.0:1788573563.867977:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573563.867983:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573563.867989:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573563.867994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573563.868005:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569729792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573563.868017:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573563.868022:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573563.868024:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573563.868030:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10317 00000100:00080000:0.0:1788573563.868034:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573563.868037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573563/real 1788573563] req@ffff920a48353480 x1875444569729536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573563.868042:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573563.868053:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573563.868061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569729536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573563.868071:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569729920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573563.868072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573563.868080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573563.868087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569729536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573563.868093:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573563.868098:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573563.868100:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573563.868106:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10317 00000100:00080000:0.0:1788573563.868110:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573563.868114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573563.868119:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573563.868126:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573563.868138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573563.868143:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569730048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573563.868150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573563/real 1788573563] req@ffff920a48350380 x1875444569729664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573563.868166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569729664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573563.868172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573563.868176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573563.868179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573563.868183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573563.868194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573563.868200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573563/real 1788573563] req@ffff920a48350700 x1875444569729792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573563.868207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569729792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573563.868212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573563.868214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573563.868216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573563.868220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573563.868231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573563.868237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573563/real 1788573563] req@ffff920a48350a80 x1875444569729920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573563.868244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569729920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573563.868249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573563.868251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573563.868253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569729920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573563.868273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573563.868285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573563.868292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573563/real 1788573563] req@ffff920a48351500 x1875444569730048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573563.868298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569730048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573563.868303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573563.868306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573563.868308:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573564.315760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573564.891724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573565.339796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573565.915709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573566.363757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573566.939747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573567.387783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573567.963697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573568.411750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573568.987730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573568.987730:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573568.987742:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573568.987746:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573568.987757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10322 00000100:00080000:3.0:1788573568.987769:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573568.987777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573568.987820:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569730176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573568.987844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573568.987847:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573568.987849:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573568.987852:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10322 00000100:00080000:3.0:1788573568.987855:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573568.987858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573568.987869:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573568.987881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569730176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573568.987882:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569730304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573568.987896:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573568.987898:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573568.987899:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573568.987902:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10322 00000100:00080000:3.0:1788573568.987904:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573568.987906:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573568.987909:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573568.987918:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569730432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573568.987926:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573568.987928:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573568.987930:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573568.987932:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10322 00000100:00080000:3.0:1788573568.987935:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573568.987938:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573568.987942:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573568.987949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569730560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573568.987956:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573568.987959:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573568.987961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573568.987964:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10322 00000100:00080000:3.0:1788573568.987966:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573568.987968:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573568.987971:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573568.987976:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573568.987981:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569730688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573568.988017:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573568/real 1788573568] req@ffff920a529b2a00 x1875444569730176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573568.988032:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569730176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573568.988041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573568.988044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573568.988047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569730176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573568.988070:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573568.988087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573568.988095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573568/real 1788573568] req@ffff920a529b3480 x1875444569730304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573568.988103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569730304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573568.988108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573568.988110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573568.988112:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573568.988116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573568.988127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573568.988133:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573568/real 1788573568] req@ffff920a529b2300 x1875444569730432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573568.988141:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569730432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573568.988146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573568.988148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573568.988150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573568.988153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573568.988163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573568.988169:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573568/real 1788573568] req@ffff920a529b1c00 x1875444569730560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573568.988176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569730560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573568.988181:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573568.988183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573568.988184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573568.988188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573568.988197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573568.988202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573568/real 1788573568] req@ffff920a529b0700 x1875444569730688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573568.988215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569730688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573593 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573568.988220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573568.988221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573568.988223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573569.435803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573570.011717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573570.459819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573571.035763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573571.483694:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573572.059721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573572.507764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573573.083737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573573.531726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573574.107715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573574.107715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573574.107726:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573574.107731:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.107742:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10327 00000100:00080000:0.0:1788573574.107753:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.107761:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573574.107798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569730816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.107826:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.107830:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.107832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.107837:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10327 00000100:00080000:0.0:1788573574.107842:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.107847:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573574.107858:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573574.107862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569730816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573574.107889:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569730944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.107903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.107907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573574.107909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.107914:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10327 00000100:00080000:0.0:1788573574.107919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.107924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573574.107930:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573574.107931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573574.107949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569731072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573574.107961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573574.107965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573574.107967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573574.107971:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff920a48353b80 x1875444569730816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573574.107976:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10327 00000100:00080000:0.0:1788573574.107983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.107993:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573574.108001:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569730816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573574.108002:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573574.108018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573574.108025:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569731200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573574.108026:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573574.108035:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569730816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573574.108051:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573574.108066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730944:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573574.108068:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573574.108076:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573574.108083:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10327 00000100:00080000:0.0:1788573574.108087:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573574.108091:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573574.108097:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573574.108112:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569731328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573574.108117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573574.108144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff920a48353800 x1875444569730944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573574.108161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569730944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573574.108171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573574.108175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573574.108179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569730944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573574.108187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573574.108209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573574.108220:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff920a48351c00 x1875444569731072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573574.108234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569731072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573574.108242:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573574.108246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573574.108249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573574.108255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573574.108274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573574.108284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff920a48351500 x1875444569731200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573574.108313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569731200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573574.108321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573574.108325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573574.108328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573574.108347:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573574.108368:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573574.108380:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573574/real 1788573574] req@ffff920a48350a80 x1875444569731328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573574.108403:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569731328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573574.108411:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573574.108414:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573574.108418:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573574.555780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573575.131732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573575.579805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573576.155747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573576.603738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573577.179707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573577.627741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573578.203702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573578.651758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573579.227684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573579.227721:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573579.227729:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573579.227733:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573579.227743:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10332 00000100:00080000:3.0:1788573579.227753:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573579.227761:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573579.227799:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569731456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573579.227825:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573579.227830:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573579.227832:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573579.227836:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10332 00000100:00080000:3.0:1788573579.227840:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573579.227845:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573579.227856:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573579.227872:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569731584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573579.227874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569731456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573579.227892:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573579.227896:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573579.227898:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573579.227903:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10332 00000100:00080000:3.0:1788573579.227907:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573579.227911:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573579.227917:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573579.227928:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569731712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573579.227936:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573579.227941:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573579.227943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573579.227949:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10332 00000800:00000100:1.0:1788573579.227950:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573579.227957:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573579.227964:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573579.227971:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573579.227985:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569731840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573579.227995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff920a529b2d80 x1875444569731456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573579.228000:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573579.228011:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573579.228015:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573579.228026:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10332 00000100:00080000:3.0:1788573579.228033:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573579.228040:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573579.228046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569731456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788573579.228048:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573579.228064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573579.228069:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569731968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573579.228073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573579.228082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569731456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573579.228114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573579.228142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573579.228156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff920a529b3100 x1875444569731584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573579.228170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569731584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573579.228180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573579.228184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573579.228188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573579.228195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573579.228216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573579.228229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff920a529b0000 x1875444569731712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573579.228242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569731712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573579.228259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573579.228263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573579.228266:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573579.228273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573579.228307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573579.228319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff920a529b1f80 x1875444569731840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573579.228332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569731840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573579.228340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573579.228343:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573579.228347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573579.228353:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573579.228373:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573579.228383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573579/real 1788573579] req@ffff920a529b0e00 x1875444569731968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573579.228395:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569731968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573579.228402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573579.228405:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573579.228409:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569731968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573579.675753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573580.251682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573580.699737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573581.275717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573581.723796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573582.299716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573582.747814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573583.323716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573583.771777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573584.347696:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573584.347709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573584.347720:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573584.347725:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573584.347735:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10337 00000100:00080000:3.0:1788573584.347754:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573584.347768:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573584.347806:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569732096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573584.347835:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573584.347840:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573584.347842:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573584.347847:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10337 00000100:00080000:3.0:1788573584.347853:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573584.347859:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573584.347871:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573584.347878:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569732096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573584.347902:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569732224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573584.347916:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573584.347920:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573584.347922:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573584.347927:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10337 00000100:00080000:3.0:1788573584.347931:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573584.347936:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573584.347941:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573584.347944:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573584.347959:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569732352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573584.347970:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573584.347975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff920a529b3b80 x1875444569732096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.347976:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573584.347980:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573584.347991:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10337 00000100:00080000:3.0:1788573584.347997:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573584.348001:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569732096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573584.348004:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573584.348017:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573584.348021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:3.0:1788573584.348023:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573584.348024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569732096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573584.348042:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569732480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573584.348051:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732224:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573584.348070:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573584.348075:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788573584.348079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573584.348080:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573584.348088:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10337 00000100:00080000:3.0:1788573584.348093:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573584.348096:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573584.348101:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573584.348105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff920a529b0380 x1875444569732224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573584.348116:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569732608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573584.348120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569732224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573584.348146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573584.348152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573584.348157:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573584.348174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573584.348202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573584.348216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff920a529b1180 x1875444569732352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573584.348230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569732352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573584.348238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573584.348242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573584.348245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573584.348257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573584.348274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573584.348282:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff920a529b1500 x1875444569732480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573584.348289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569732480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573584.348295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573584.348297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573584.348299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573584.348302:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573584.348312:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573584.348318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573584/real 1788573584] req@ffff920a529b3b80 x1875444569732608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573584.348325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569732608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573584.348330:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573584.348332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573584.348334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573584.795805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573585.371683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573585.819727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573586.395732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573586.843754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573587.419655:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573587.867709:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573588.443725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573588.891738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573589.467715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573589.467730:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.467739:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573589.467745:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.467755:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10342 00000100:00080000:0.0:1788573589.467766:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.467776:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573589.467817:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569732736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.467850:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.467854:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573589.467856:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.467862:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10342 00000100:00080000:0.0:1788573589.467866:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.467873:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573589.467882:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573589.467901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569732864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573589.467906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569732736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573589.467919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.467925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573589.467927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.467934:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10342 00000100:00080000:0.0:1788573589.467938:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.467942:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573589.467947:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573589.467961:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569732992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.467970:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.467974:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573589.467976:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.467981:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10342 00000100:00080000:0.0:1788573589.467988:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.467992:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573589.467998:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573589.468006:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573589.468014:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569733120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573589.468028:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573589.468033:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573589.468036:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573589.468042:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10342 00000100:00080000:0.0:1788573589.468046:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573589.468050:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573589.468056:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573589.468066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff920a48350700 x1875444569732736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573589.468069:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569733248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573589.468148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569732736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573589.468165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573589.468170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573589.468176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569732736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573589.468209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573589.468238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573589.468261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff920a48350380 x1875444569732864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573589.468283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569732864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573589.468293:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573589.468299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573589.468307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573589.468318:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573589.468346:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573589.468363:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff920a48353480 x1875444569732992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573589.468378:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569732992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573589.468393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573589.468397:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573589.468401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569732992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573589.468408:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573589.468431:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573589.468442:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff920a48350a80 x1875444569733120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573589.468455:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569733120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573589.468464:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573589.468480:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573589.468484:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573589.468491:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573589.468512:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573589.468524:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573589/real 1788573589] req@ffff920a48351500 x1875444569733248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573589.468551:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569733248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573589.468561:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573589.468564:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573589.468568:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573589.915764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573590.491686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573590.939812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573591.515714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573591.963751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573592.539726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573592.987754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573593.563733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573594.011805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573594.587744:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573594.587759:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1788573594.587760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573594.587766:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573594.587784:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10347 00000100:00080000:0.0:1788573594.587798:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573594.587807:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573594.587860:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569733376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573594.587892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573594.587896:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573594.587898:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573594.587903:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10347 00000100:00080000:0.0:1788573594.587907:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573594.587911:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573594.587924:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573594.587938:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569733504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573594.587947:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569733376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573594.587963:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573594.587968:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573594.587970:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573594.587976:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10347 00000100:00080000:0.0:1788573594.587980:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573594.587985:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573594.587990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573594.588003:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569733632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573594.588020:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573594.588024:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573594.588026:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573594.588031:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573594.588032:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10347 00000100:00080000:0.0:1788573594.588039:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573594.588043:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573594.588050:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573594.588064:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569733760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573594.588073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573594] req@ffff920a48351c00 x1875444569733376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573594.588076:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573594.588085:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573594.588089:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573594.588095:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10347 00000100:00080000:0.0:1788573594.588099:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573594.588102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569733376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573594.588106:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573594.588111:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573594.588124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573594.588126:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569733888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573594.588131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573594.588135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569733376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573594.588161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573594.588182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573594.588196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573594] req@ffff920a48353800 x1875444569733504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573594.588209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569733504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573594.588228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573594.588232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573594.588237:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573594.588243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573594.588286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573594.588314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573594] req@ffff920a48353b80 x1875444569733632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573594.588330:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569733632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573594.588339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573594.588343:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573594.588347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573594.588353:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573594.588379:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573594.588393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573594] req@ffff920a48351500 x1875444569733760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573594.588407:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569733760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573594.588423:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573594.588425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573594.588428:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573594.588433:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573594.588450:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573594.588482:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573594/real 1788573594] req@ffff920a48350a80 x1875444569733888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573594.588491:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569733888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573594.588503:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573594.588505:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573594.588508:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569733888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573595.035810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573595.611685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573596.059743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573596.635693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573597.083719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573597.659705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573598.107749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573598.683700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573599.131804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573599.707702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573599.707732:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573599.707738:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573599.707743:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573599.707754:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10353 00000100:00080000:0.0:1788573599.707766:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573599.707773:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573599.707829:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569734016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573599.707865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573599.707875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573599.707877:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573599.707882:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10353 00000100:00080000:0.0:1788573599.707887:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573599.707892:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573599.707905:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573599.707914:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569734016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573599.707925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569734144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573599.707938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573599.707942:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573599.707944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573599.707949:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10353 00000100:00080000:0.0:1788573599.707954:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573599.707959:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573599.707965:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573599.707978:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569734272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573599.707988:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573599.708001:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573599.708004:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573599.708006:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573599.708011:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10353 00000100:00080000:0.0:1788573599.708015:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573599.708018:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573599.708023:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573599.708028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573599/real 1788573599] req@ffff920a48353480 x1875444569734016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573599.708039:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569734400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573599.708053:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573599.708053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569734016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573599.708064:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573599.708069:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573599.708078:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10353 00000100:00080000:2.0:1788573599.708079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573599.708083:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573599.708088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788573599.708091:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573599.708098:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573599.708098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569734016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573599.708111:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569734528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573599.708133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573599.708155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573599.708165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573599/real 1788573599] req@ffff920a48350380 x1875444569734144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573599.708174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569734144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573599.708180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573599.708183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573599.708185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573599.708190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573599.708202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573599.708209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573599/real 1788573599] req@ffff920a48350700 x1875444569734272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573599.708217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569734272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573599.708222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573599.708224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573599.708226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573599.708229:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573599.708241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573599.708247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573599/real 1788573599] req@ffff920a48350a80 x1875444569734400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573599.708254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569734400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573599.708260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573599.708261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573599.708263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573599.708266:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573599.708276:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573599.708282:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573599/real 1788573599] req@ffff920a48351500 x1875444569734528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573599.708290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569734528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573599.708295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573599.708297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573599.708299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573600.155734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573600.731737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573601.179754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573601.755715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573602.203827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573602.779744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573603.227735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573603.803766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573604.251747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573604.827679:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573604.827685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573604.827689:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573604.827694:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573604.827704:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10358 00000100:00080000:0.0:1788573604.827714:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573604.827722:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573604.827758:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569734656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573604.827784:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573604.827787:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573604.827788:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573604.827792:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10358 00000100:00080000:0.0:1788573604.827795:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573604.827798:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573604.827809:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573604.827824:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569734784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573604.827831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569734656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573604.827843:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573604.827848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573604.827850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573604.827855:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10358 00000100:00080000:0.0:1788573604.827858:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573604.827861:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573604.827866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573604.827877:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569734912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573604.827885:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573604.827888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573604.827897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573604.827899:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573604.827904:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10358 00000100:00080000:0.0:1788573604.827907:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573604.827910:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573604.827915:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573604.827923:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569735040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573604.827934:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573604.827937:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573604.827939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573604.827942:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10358 00000100:00080000:0.0:1788573604.827945:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573604.827947:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573604.827951:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573604.827952:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573604/real 1788573604] req@ffff920a48353b80 x1875444569734656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573604.827961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569735168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573604.827973:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569734656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573604.827989:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573604.827993:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573604.827996:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569734656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573604.828015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573604.828034:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573604.828043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573604/real 1788573604] req@ffff920a48353800 x1875444569734784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573604.828051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569734784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573604.828057:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573604.828059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573604.828062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573604.828065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573604.828076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573604.828081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573604/real 1788573604] req@ffff920a48351c00 x1875444569734912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573604.828088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569734912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573604.828094:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573604.828096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573604.828098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569734912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573604.828101:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573604.828111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573604.828116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573604/real 1788573604] req@ffff920a48351500 x1875444569735040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573604.828123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569735040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573604.828128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573604.828129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573604.828132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573604.828136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573604.828154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573604.828160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573604/real 1788573604] req@ffff920a48350a80 x1875444569735168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573604.828167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569735168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573604.828173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573604.828175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573604.828177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573605.275737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573605.851736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573606.299835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573606.875729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573607.323805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573607.899736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573608.347785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573608.923729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573609.371776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573609.947719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573609.947731:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573609.947739:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573609.947744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573609.947755:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10363 00000100:00080000:1.0:1788573609.947764:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573609.947772:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573609.947808:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569735296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573609.947842:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573609.947846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573609.947848:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573609.947853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10363 00000100:00080000:1.0:1788573609.947857:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573609.947861:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573609.947871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573609.947885:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569735424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573609.947897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573609.947901:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573609.947903:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573609.947908:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10363 00000100:00080000:1.0:1788573609.947912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573609.947916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573609.947921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573609.947936:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569735552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573609.947938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569735296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573609.947965:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573609.947969:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573609.947971:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573609.947976:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10363 00000100:00080000:1.0:1788573609.947981:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573609.947985:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573609.947991:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573609.948003:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569735680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573609.948013:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573609.948017:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573609.948018:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573609.948022:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10363 00000100:00080000:1.0:1788573609.948027:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573609.948032:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573609.948037:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573609.948049:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569735808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573609.948088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573609.948144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573609.948160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573609.948184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573610.395777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573610.971799:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573611.419787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573611.995760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573612.443776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573613.019720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788573613.019835:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788573613.019873:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573613.019887:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573613.019919:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573613.019931:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573613.019949:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573613.019957:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788573613.019966:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573609/real 1788573613] req@ffff920a52891f80 x1875444569735552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573613.020010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569735552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573613.020024:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573613.020038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573613.020043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573613.020052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573609/real 1788573613] req@ffff920a52892a00 x1875444569735680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573613.020065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569735680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573613.020072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573613.020076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573613.020079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573613.020086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573609/real 1788573613] req@ffff920a52893800 x1875444569735808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573613.020103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569735808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573613.020111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573613.020114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573613.020117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735808:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573613.020125:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573609/real 1788573613] req@ffff920a52893480 x1875444569735296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573613.020137:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569735296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573613.020145:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573613.020149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573613.020153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569735296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573613.020161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573609/real 1788573613] req@ffff920a52890380 x1875444569735424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573613.020174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569735424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573634 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573613.020183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573613.020186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573613.020189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569735424:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788573613.467758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573614.043803:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573614.491799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573615.067712:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.067723:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573615.067728:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.067740:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10368 00000100:00080000:0.0:1788573615.067751:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.067760:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573615.067795:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788573615.067798:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569735936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573615.067831:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.067834:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573615.067836:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.067843:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10368 00000100:00080000:0.0:1788573615.067847:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.067851:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573615.067860:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573615.067873:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569736064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573615.067883:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569735936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573615.067884:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.067890:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573615.067894:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.067899:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10368 00000100:00080000:0.0:1788573615.067903:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.067906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573615.067912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573615.067926:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569736192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573615.067936:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.067940:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573615.067943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.067947:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10368 00000100:00080000:0.0:1788573615.067951:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.067957:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573615.067957:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573615.067966:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573615.067982:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569736320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573615.067992:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573615.067996:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573615.067998:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573615.068004:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10368 00000100:00100000:1.0:1788573615.068006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff920a48350700 x1875444569735936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573615.068009:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573615.068018:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573615.068026:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573615.068038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569735936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573615.068043:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569736448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573615.068071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573615.068080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573615.068093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569735936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573615.068122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573615.068148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573615.068175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff920a48350380 x1875444569736064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573615.068189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569736064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573615.068198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573615.068201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573615.068205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573615.068211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573615.068232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573615.068244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff920a48353480 x1875444569736192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573615.068255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569736192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573615.068263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573615.068266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573615.068269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573615.068276:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573615.068296:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573615.068307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff920a48350a80 x1875444569736320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573615.068319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569736320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573615.068327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573615.068331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573615.068334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573615.068340:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573615.068363:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573615.068375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573615/real 1788573615] req@ffff920a48351500 x1875444569736448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573615.068387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569736448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573615.068395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573615.068398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573615.068401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573615.515757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573616.091779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573616.539777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573617.115685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573617.563771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573618.139726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573618.587775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573619.163706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573619.611761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573620.187697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573620.187701:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.187708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573620.187713:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.187724:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10373 00000100:00080000:0.0:1788573620.187735:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.187744:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573620.187778:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569736576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.187799:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.187802:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573620.187803:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.187806:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10373 00000100:00080000:0.0:1788573620.187810:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.187813:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573620.187824:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573620.187835:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569736704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573620.187839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569736576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573620.187846:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.187855:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573620.187859:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.187863:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10373 00000100:00080000:0.0:1788573620.187873:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.187878:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573620.187882:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573620.187891:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569736832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.187904:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.187907:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573620.187908:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.187911:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10373 00000100:00080000:0.0:1788573620.187921:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.187924:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573620.187929:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573620.187938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573620.187940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569736960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573620.187950:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573620.187953:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573620.187955:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573620.187960:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10373 00000100:00080000:0.0:1788573620.187962:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573620.187966:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573620.187969:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573620.187981:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569737088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573620.187992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff920a48351c00 x1875444569736576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.188014:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569736576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.188023:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.188028:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573620.188032:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569736576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573620.188066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573620.188107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573620.188123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff920a48353800 x1875444569736704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.188138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569736704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.188146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.188150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573620.188153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573620.188160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573620.188182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573620.188194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff920a48353b80 x1875444569736832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.188212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569736832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.188220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.188223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573620.188227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573620.188233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573620.188255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573620.188265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff920a48351500 x1875444569736960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.188273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569736960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.188279:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.188281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573620.188283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569736960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573620.188287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573620.188299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573620.188306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573620/real 1788573620] req@ffff920a48350a80 x1875444569737088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573620.188313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569737088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573620.188328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573620.188331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573620.188334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573620.635802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573621.211730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573621.659765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573622.235715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573622.683786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573623.259674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573623.707805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573624.283709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573624.731743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573625.307760:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.307772:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573625.307777:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.307789:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10378 00000100:00080000:2.0:1788573625.307801:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.307812:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573625.307843:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788573625.307858:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569737216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.307887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.307891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573625.307895:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.307899:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10378 00000100:00080000:2.0:1788573625.307902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.307906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573625.307917:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573625.307918:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569737216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573625.307937:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569737344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.307948:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.307959:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573625.307961:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.307968:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10378 00000100:00080000:2.0:1788573625.307988:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.307992:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573625.307997:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573625.308000:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573625.308041:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569737472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.308052:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.308056:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573625.308060:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.308068:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10378 00000100:00080000:2.0:1788573625.308075:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.308081:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573625.308086:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573625.308094:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569737600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573625.308101:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573625.308104:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573625.308105:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573625.308107:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10378 00000100:00080000:2.0:1788573625.308109:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573625.308112:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573625.308116:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573625.308124:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569737728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573625.308153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff920a60ac4700 x1875444569737216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.308179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569737216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.308191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.308197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573625.308201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569737216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573625.308364:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573625.308387:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573625.308398:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff920a60ac4a80 x1875444569737344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.308411:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569737344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.308457:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.308462:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573625.308466:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573625.308473:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573625.308489:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573625.308497:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff920a60ac5500 x1875444569737472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.308505:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569737472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.308511:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.308513:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573625.308515:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573625.308519:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573625.308530:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573625.308535:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff920a60ac6300 x1875444569737600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.308542:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569737600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.308547:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.308549:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573625.308552:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573625.308555:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573625.308565:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573625.308570:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573625/real 1788573625] req@ffff920a60ac7100 x1875444569737728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573625.308576:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569737728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573625.308581:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573625.308583:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573625.308605:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573625.755786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573626.331700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573626.779806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573627.355668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573627.803807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573628.379679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573628.827769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573629.403718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573629.851787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573630.427718:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573630.427721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573630.427730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573630.427736:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.427746:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10383 00000100:00080000:0.0:1788573630.427757:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.427764:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573630.427808:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569737856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.427837:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.427841:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573630.427851:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.427861:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10383 00000100:00080000:0.0:1788573630.427865:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.427871:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573630.427884:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573630.427900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569737856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573630.427903:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569737984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.427919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.427924:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573630.427926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.427932:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10383 00000100:00080000:0.0:1788573630.427936:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.427942:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573630.427962:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573630.427978:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569738112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573630.427989:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.427993:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573630.427996:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573630.428002:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573630.428005:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10383 00000100:00080000:0.0:1788573630.428012:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573630.428018:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573630.428024:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573630.428041:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569738240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573630.428053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff920a48353480 x1875444569737856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573630.428054:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573630.428063:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573630.428068:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573630.428078:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10383 00000100:00080000:0.0:1788573630.428084:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573630.428089:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569737856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573630.428092:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573630.428102:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573630.428108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788573630.428118:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569738368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573630.428128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573630.428136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569737856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573630.428166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573630.428195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573630.428209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff920a48350380 x1875444569737984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.428222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569737984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.428231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.428234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573630.428238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569737984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573630.428244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573630.428262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573630.428273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff920a48350700 x1875444569738112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.428284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569738112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.428292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.428296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573630.428299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573630.428304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573630.428322:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573630.428333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff920a48350a80 x1875444569738240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.428344:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569738240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.428351:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.428354:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573630.428357:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573630.428363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573630.428380:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573630.428390:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573630/real 1788573630] req@ffff920a48351500 x1875444569738368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573630.428401:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569738368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573630.428409:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573630.428412:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573630.428415:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573630.875786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573631.451722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573631.899787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573632.475699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573632.923730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573633.499745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573633.947787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573634.523665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573634.971727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573635.547696:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573635.547705:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573635.547709:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573635.547717:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10388 00000100:00100000:1.0:1788573635.547723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573635.547727:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573635.547735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573635.547772:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569738496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573635.547800:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573635.547804:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573635.547807:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573635.547812:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10388 00000100:00080000:0.0:1788573635.547817:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573635.547823:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573635.547834:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569738496:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573635.547835:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573635.547855:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569738624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573635.547882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573635.547887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573635.547889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573635.547894:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10388 00000100:00080000:0.0:1788573635.547900:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573635.547905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788573635.547911:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573635.547916:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573635.547934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569738752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573635.547944:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff920a48353b80 x1875444569738496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573635.547947:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573635.547957:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573635.547960:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573635.547965:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10388 00000100:00100000:3.0:1788573635.547967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569738496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573635.547970:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573635.547978:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788573635.547981:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788573635.547984:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573635.547987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573635.547991:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569738496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573635.547999:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569738880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573635.548007:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573635.548009:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573635.548010:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573635.548013:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738624:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573635.548017:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10388 00000100:00080000:0.0:1788573635.548021:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573635.548024:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573635.548027:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573635.548035:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569739008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573635.548039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573635.548053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff920a48353800 x1875444569738624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.548063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569738624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.548070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.548073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573635.548076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573635.548080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573635.548093:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573635.548099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff920a48351c00 x1875444569738752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.548106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569738752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.548111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.548112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573635.548115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573635.548121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573635.548142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573635.548154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff920a48351500 x1875444569738880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.548167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569738880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.548176:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.548179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573635.548183:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569738880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573635.548199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573635.548221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573635.548233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573635/real 1788573635] req@ffff920a48350a80 x1875444569739008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573635.548241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569739008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573635.548246:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573635.548248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573635.548250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573635.995753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573636.571673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573637.019770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573637.595711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573638.043780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573638.619676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573639.067730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573639.643750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573640.091717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573640.667697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573640.667726:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573640.667733:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573640.667737:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573640.667747:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10394 00000100:00080000:2.0:1788573640.667757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573640.667764:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573640.667815:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569739136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573640.667846:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573640.667851:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573640.667853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573640.667858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10394 00000100:00080000:2.0:1788573640.667862:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573640.667867:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573640.667876:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573640.667891:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569739264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573640.667896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569739136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573640.667909:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573640.667913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573640.667915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573640.667920:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10394 00000100:00080000:2.0:1788573640.667925:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573640.667929:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573640.667944:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573640.667964:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569739392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573640.667970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573640.667979:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573640.667984:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573640.667986:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573640.667991:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10394 00000100:00080000:2.0:1788573640.667996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573640.668001:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573640.668008:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573640.668012:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573640/real 1788573640] req@ffff920a60ac6a00 x1875444569739136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573640.668022:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569739520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573640.668036:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573640.668039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569739136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573640.668041:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573640.668045:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573640.668055:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10394 00000100:00080000:2.0:1788573640.668062:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573640.668063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573640.668069:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788573640.668073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:2.0:1788573640.668077:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573640.668080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569739136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573640.668092:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569739648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573640.668112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573640.668141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573640.668157:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573640/real 1788573640] req@ffff920a60ac5180 x1875444569739264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573640.668171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569739264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573640.668181:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573640.668184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573640.668188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573640.668194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573640.668216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573640.668229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573640/real 1788573640] req@ffff920a60ac7100 x1875444569739392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573640.668242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569739392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573640.668257:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573640.668260:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573640.668264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573640.668270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573640.668292:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573640.668304:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573640/real 1788573640] req@ffff920a60ac6300 x1875444569739520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573640.668316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569739520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573640.668324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573640.668328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573640.668331:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573640.668338:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573640.668359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573640.668371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573640/real 1788573640] req@ffff920a60ac5500 x1875444569739648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573640.668384:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569739648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573665 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573640.668393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573640.668396:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573640.668400:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573641.115787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573641.691719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573642.139752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573642.715693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573643.163774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573643.739734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573644.187706:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573644.763708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573645.211775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573645.787688:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573645.787696:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573645.787700:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573645.787708:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10399 00000100:00080000:0.0:1788573645.787718:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573645.787743:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573645.787746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788573645.787778:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569739776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573645.787801:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573645.787804:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573645.787805:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573645.787808:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10399 00000100:00080000:0.0:1788573645.787811:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573645.787814:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573645.787822:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573645.787832:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569739904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573645.787844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573645.787849:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573645.787850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573645.787850:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569739776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573645.787858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10399 00000100:00080000:0.0:1788573645.787861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573645.787863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573645.787866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573645.787875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569740032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573645.787881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573645.787884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573645.787885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573645.787888:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10399 00000100:00080000:0.0:1788573645.787890:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573645.787893:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573645.787896:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573645.787902:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569740160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573645.787908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573645.787910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573645.787912:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788573645.787915:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573645.787918:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10399 00000100:00080000:0.0:1788573645.787922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573645.787926:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573645.787929:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573645.787937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569740288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573645.787954:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573645/real 1788573645] req@ffff920a48350700 x1875444569739776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573645.787969:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569739776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573645.787984:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573645.787987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573645.787991:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569739776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573645.788013:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573645.788030:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573645.788038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573645/real 1788573645] req@ffff920a48350380 x1875444569739904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573645.788046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569739904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573645.788052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573645.788054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573645.788056:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569739904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573645.788059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573645.788069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573645.788076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573645/real 1788573645] req@ffff920a48353480 x1875444569740032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573645.788083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569740032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573645.788088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573645.788090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573645.788091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573645.788095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573645.788105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573645.788110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573645/real 1788573645] req@ffff920a48350a80 x1875444569740160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573645.788117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569740160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573645.788121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573645.788123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573645.788125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573645.788128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573645.788137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573645.788143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573645/real 1788573645] req@ffff920a48351500 x1875444569740288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573645.788149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569740288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573670 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573645.788154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573645.788156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573645.788158:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573646.235774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573646.811728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573647.259807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573647.835735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573648.283875:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573648.859729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573649.307774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573649.883754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573650.331736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573650.907695:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573650.907698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573650.907706:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573650.907711:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573650.907719:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10404 00000100:00080000:0.0:1788573650.907729:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573650.907735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573650.907767:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569740416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573650.907790:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573650.907794:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573650.907795:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573650.907799:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10404 00000100:00080000:0.0:1788573650.907802:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573650.907805:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573650.907813:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573650.907822:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569740416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573650.907824:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569740544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573650.907842:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573650.907845:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573650.907846:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573650.907849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10404 00000100:00080000:0.0:1788573650.907852:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573650.907855:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573650.907858:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573650.907866:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569740672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573650.907872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573650.907875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573650.907876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573650.907879:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10404 00000100:00080000:0.0:1788573650.907882:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573650.907886:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573650.907888:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573650.907893:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573650.907903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569740800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573650.907910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573650.907913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573650.907914:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573650.907919:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10404 00000100:00100000:3.0:1788573650.907921:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573650/real 1788573650] req@ffff920a48351c00 x1875444569740416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573650.907923:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573650.907929:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573650.907936:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573650.907948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569740928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573650.907948:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569740416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573650.907982:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573650.907987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573650.907990:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569740416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573650.908016:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573650.908032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573650.908040:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573650/real 1788573650] req@ffff920a48353800 x1875444569740544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573650.908048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569740544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573650.908054:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573650.908056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573650.908058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573650.908062:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573650.908073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573650.908079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573650/real 1788573650] req@ffff920a48353b80 x1875444569740672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573650.908086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569740672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573650.908091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573650.908093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573650.908095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573650.908098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573650.908108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573650.908114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573650/real 1788573650] req@ffff920a48351500 x1875444569740800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573650.908121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569740800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573650.908126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573650.908128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573650.908129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573650.908133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573650.908142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573650.908148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573650/real 1788573650] req@ffff920a48350a80 x1875444569740928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573650.908155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569740928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573675 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573650.908160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573650.908161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573650.908163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569740928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573651.355739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573651.931708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573652.379787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573652.955731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573653.403835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573653.979694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573654.427723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573655.003706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573655.451774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573656.027677:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573656.027683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573656.027690:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573656.027696:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573656.027708:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10409 00000100:00080000:1.0:1788573656.027719:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573656.027727:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573656.027761:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569741056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573656.027784:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573656.027787:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573656.027788:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573656.027791:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10409 00000100:00080000:1.0:1788573656.027794:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573656.027797:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573656.027805:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573656.027813:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569741184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573656.027821:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573656.027824:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573656.027825:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573656.027830:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10409 00000100:00080000:1.0:1788573656.027832:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573656.027835:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573656.027838:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573656.027842:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569741056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573656.027847:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569741312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573656.027856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573656.027858:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573656.027859:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573656.027862:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10409 00000100:00080000:1.0:1788573656.027865:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573656.027867:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573656.027870:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573656.027878:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569741440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573656.027885:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573656.027887:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573656.027888:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573656.027890:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10409 00000100:00080000:1.0:1788573656.027894:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573656.027896:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573656.027899:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573656.027904:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573656.027908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569741568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573656.027955:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff920a52890e00 x1875444569741056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573656.027976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569741056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573656.027987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573656.027992:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573656.028001:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569741056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573656.028010:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573656.028037:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573656.028053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff920a52891180 x1875444569741184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573656.028067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569741184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573656.028082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573656.028085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573656.028087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573656.028107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573656.028123:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573656.028132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff920a52890700 x1875444569741312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573656.028144:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569741312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573656.028153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573656.028156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573656.028160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573656.028167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573656.028197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573656.028211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff920a52893100 x1875444569741440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573656.028224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569741440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573656.028232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573656.028236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573656.028240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573656.028247:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573656.028264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573656.028272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573656/real 1788573656] req@ffff920a52891880 x1875444569741568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573656.028279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569741568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573656.028285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573656.028287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573656.028289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573656.475710:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573657.051713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573657.499752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573658.075677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573658.523705:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573659.099672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573659.547743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573660.123679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573660.571754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573661.147692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573661.147697:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573661.147706:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573661.147710:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573661.147721:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10414 00000100:00080000:2.0:1788573661.147733:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573661.147744:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573661.147781:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569741696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573661.147801:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573661.147805:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573661.147807:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573661.147811:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10414 00000100:00080000:2.0:1788573661.147815:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573661.147820:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573661.147825:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569741696:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788573661.147831:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573661.147846:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569741824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573661.147858:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573661.147864:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573661.147867:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573661.147874:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10414 00000100:00080000:2.0:1788573661.147880:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573661.147886:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573661.147895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573661.147899:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573661.147916:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569741952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573661.147930:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573661.147935:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573661.147938:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573661.147946:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10414 00000100:00100000:0.0:1788573661.147948:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573661] req@ffff920a60ac4a80 x1875444569741696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573661.147952:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573661.147963:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573661.147972:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573661.147979:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569741696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573661.147991:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569742080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573661.147996:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573661.148006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788573661.148009:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573661.148015:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569741696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788573661.148016:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573661.148021:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573661.148028:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10414 00000100:00080000:2.0:1788573661.148033:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573661.148038:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573661.148045:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573661.148048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573661.148062:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569742208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573661.148082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573661.148101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573661] req@ffff920a60ac4700 x1875444569741824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573661.148117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569741824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573661.148127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573661.148143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573661.148151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573661.148159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573661.148183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573661.148196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573661] req@ffff920a60ac4380 x1875444569741952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573661.148211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569741952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573661.148221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573661.148225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573661.148229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569741952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573661.148237:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573661.148261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573661.148275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573661] req@ffff920a60ac4000 x1875444569742080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573661.148297:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569742080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573661.148308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573661.148313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573661.148319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573661.148338:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573661.148362:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573661.148375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573661/real 1788573661] req@ffff920a60ac7800 x1875444569742208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573661.148388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569742208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573661.148431:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573661.148435:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573661.148439:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573661.595758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573662.171735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573662.619826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573663.195720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573663.643703:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573664.219750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573664.667791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573665.243667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573665.691720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573666.267757:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573666.267767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573666.267780:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573666.267785:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.267799:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10419 00000100:00080000:2.0:1788573666.267812:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.267829:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573666.267876:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569742336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.267917:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.267921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573666.267923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.267927:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10419 00000100:00080000:2.0:1788573666.267931:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.267936:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573666.267948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573666.267958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569742336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573666.267968:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569742464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.267979:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.267983:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573666.267985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.267989:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10419 00000100:00080000:2.0:1788573666.267991:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.267995:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573666.268001:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573666.268017:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569742592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573666.268029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573666.268029:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.268037:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573666.268041:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.268048:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10419 00000100:00080000:2.0:1788573666.268056:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.268062:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573666.268068:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573666.268076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff920a60ac5c00 x1875444569742336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573666.268087:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569742720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573666.268152:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573666.268159:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573666.268161:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573666.268165:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10419 00000100:00080000:2.0:1788573666.268170:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573666.268172:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573666.268176:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573666.268179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569742336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573666.268189:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569742848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573666.268195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.268208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573666.268212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569742336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573666.268239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573666.268261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573666.268289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff920a60ac4e00 x1875444569742464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.268299:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569742464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.268306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.268308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573666.268311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573666.268316:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573666.268332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573666.268339:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff920a60ac7800 x1875444569742592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.268347:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569742592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.268357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.268360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573666.268362:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573666.268366:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573666.268377:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573666.268383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff920a60ac4000 x1875444569742720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.268390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569742720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.268396:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.268398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573666.268399:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573666.268403:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573666.268413:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573666.268418:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573666/real 1788573666] req@ffff920a60ac4380 x1875444569742848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573666.268425:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569742848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573666.268430:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573666.268432:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573666.268434:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569742848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573666.715852:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573667.291724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573667.739762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573668.315721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573668.763756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573669.339668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573669.787752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573670.363727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573670.811803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573671.387682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573671.387688:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.387699:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573671.387705:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.387718:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10424 00000100:00080000:0.0:1788573671.387733:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.387743:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573671.387801:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569742976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.387828:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.387831:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573671.387833:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.387836:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10424 00000100:00080000:0.0:1788573671.387840:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.387844:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573671.387854:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573671.387868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569743104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573671.387879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569742976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573671.387882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573671.387897:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573671.387899:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.387906:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10424 00000100:00080000:0.0:1788573671.387912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.387918:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573671.387925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573671.387942:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569743232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.387954:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788573671.387954:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573671.387962:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573671.387964:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.387970:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10424 00000100:00080000:0.0:1788573671.387976:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.387980:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573671.387987:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573671.388004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff920a48353480 x1875444569742976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573671.388005:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569743360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573671.388043:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573671.388048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569742976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573671.388050:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573671.388056:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573671.388074:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10424 00000100:00080000:0.0:1788573671.388082:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573671.388085:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573671.388092:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573671.388093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573671.388104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573671.388108:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569743488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573671.388119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569742976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573671.388171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573671.388195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573671.388205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff920a48350380 x1875444569743104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573671.388214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569743104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573671.388221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573671.388223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573671.388226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573671.388230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573671.388242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573671.388249:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff920a48350700 x1875444569743232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573671.388256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569743232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573671.388261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573671.388263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573671.388265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573671.388269:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573671.388281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573671.388286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff920a48350a80 x1875444569743360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573671.388293:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569743360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573671.388299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573671.388301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573671.388303:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573671.388314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573671.388332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573671.388343:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573671/real 1788573671] req@ffff920a48351500 x1875444569743488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573671.388356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569743488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573671.388364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573671.388368:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573671.388371:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573671.835807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573672.411723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573672.859795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573673.435670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573673.883742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573674.459720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573674.907747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573675.483740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573675.931744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573676.507689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573676.507705:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573676.507713:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573676.507719:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573676.507733:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10429 00000100:00080000:1.0:1788573676.507745:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573676.507754:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573676.508160:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569743616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573676.508197:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573676.508200:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573676.508202:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573676.508205:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10429 00000100:00080000:1.0:1788573676.508209:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573676.508212:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573676.508239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569743616:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573676.508254:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573676.508314:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569743744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573676.508343:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573676.508357:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743744:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573676.508362:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573676.508365:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573676.508369:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10429 00000100:00080000:1.0:1788573676.508381:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573676.508410:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573676.508429:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573676.508446:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569743872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573676.508517:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573676.508520:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573676.508522:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573676.508540:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10429 00000100:00100000:3.0:1788573676.508542:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743872:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573676.508611:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573676.508616:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573676.508622:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573676.508632:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569744000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573676.508645:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573676.508647:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573676.508649:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573676.508651:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10429 00000100:00080000:1.0:1788573676.508653:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573676.508655:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573676.508660:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573676.508673:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569744128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573676.508679:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573676.508723:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573676.955748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573677.531760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573677.979740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573678.555807:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573679.003749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573679.579874:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788573679.579905:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788573679.579985:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788573679.580002:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573679.580063:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573679.580072:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573679.580078:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788573679.580084:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788573679.580138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573679] req@ffff920a52890700 x1875444569744128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573679.580178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569744128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573679.580190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573679.580196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573679.580200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744128:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573679.580209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573679] req@ffff920a52893100 x1875444569744000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573679.580223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569744000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573679.580231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573679.580234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573679.580238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573679.580245:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573679] req@ffff920a52891880 x1875444569743872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573679.580257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569743872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573679.580266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573679.580270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573679.580273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573679.580281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573679] req@ffff920a52892d80 x1875444569743744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573679.580294:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569743744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573679.580303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573679.580306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573679.580310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569743744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573679.580318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573676/real 1788573679] req@ffff920a52892300 x1875444569743616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573679.580332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569743616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573679.580340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573679.580344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573679.580347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569743616:192.168.202.157@tcp:250:kworker.0 00010000:00010000:0.0:1788573680.027755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573680.603752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573681.051746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573681.627701:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573681.627708:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573681.627712:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573681.627722:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10435 00000100:00080000:1.0:1788573681.627734:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573681.627742:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573681.627753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788573681.627779:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569744256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573681.627807:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573681.627811:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573681.627820:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573681.627825:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10435 00000100:00080000:1.0:1788573681.627830:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573681.627835:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573681.627845:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573681.627848:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569744256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573681.627861:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569744384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573681.627873:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573681.627877:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573681.627909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573681.627914:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10435 00000100:00080000:1.0:1788573681.627919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573681.627923:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788573681.627929:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573681.627931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573681.627951:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569744512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573681.627962:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573681.627968:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1788573681.627969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573681/real 1788573681] req@ffff920a52891180 x1875444569744256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573681.627970:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573681.627981:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10435 00000100:00080000:1.0:1788573681.627987:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573681.627994:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573681.627999:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573681.628001:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569744256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573681.628014:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569744640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573681.628015:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573681.628026:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573681.628031:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569744256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573681.628043:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573681.628046:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573681.628049:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573681.628054:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10435 00000100:00080000:1.0:1788573681.628058:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573681.628059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744384:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573681.628065:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573681.628072:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573681.628087:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569744768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573681.628096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573681.628117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573681/real 1788573681] req@ffff920a52890e00 x1875444569744384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573681.628133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569744384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573681.628142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573681.628145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573681.628149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573681.628155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573681.628176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573681.628186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573681/real 1788573681] req@ffff920a52890380 x1875444569744512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573681.628198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569744512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573681.628207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573681.628211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573681.628215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573681.628221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573681.628241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573681.628252:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573681/real 1788573681] req@ffff920a52893480 x1875444569744640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573681.628265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569744640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573681.628273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573681.628276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573681.628280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573681.628295:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573681.628314:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573681.628325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573681/real 1788573681] req@ffff920a52893800 x1875444569744768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573681.628338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569744768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573706 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573681.628347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573681.628350:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573681.628354:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569744768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573682.075773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573682.651740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573683.099688:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573683.675719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573684.123777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573684.699669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573685.147770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573685.723703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573686.171823:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573686.747672:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573686.747676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573686.747682:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573686.747686:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573686.747696:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10440 00000100:00080000:1.0:1788573686.747706:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573686.747714:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573686.747762:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569744896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573686.747785:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573686.747789:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573686.747791:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573686.747804:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10440 00000100:00080000:1.0:1788573686.747809:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573686.747813:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573686.747824:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573686.747832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569744896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573686.747837:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569745024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573686.747847:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573686.747850:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573686.747852:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573686.747856:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10440 00000100:00080000:1.0:1788573686.747859:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573686.747863:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573686.747867:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573686.747878:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569745152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573686.747888:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573686.747891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573686.747893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573686.747896:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10440 00000800:00000100:3.0:1788573686.747903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573686.747907:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573686.747910:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573686.747915:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573686.747925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569745280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573686.747934:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573686.747935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573686/real 1788573686] req@ffff920a52892a00 x1875444569744896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573686.747938:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573686.747940:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573686.747944:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10440 00000100:00080000:1.0:1788573686.747947:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573686.747950:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573686.747954:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569744896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573686.747955:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573686.747964:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569745408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573686.747973:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573686.747976:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573686.747981:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569744896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573686.748003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573686.748026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573686.748037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573686/real 1788573686] req@ffff920a52891f80 x1875444569745024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573686.748051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569745024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573686.748060:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573686.748064:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573686.748068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573686.748075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573686.748097:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573686.748108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573686/real 1788573686] req@ffff920a52891500 x1875444569745152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573686.748121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569745152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573686.748129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573686.748132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573686.748135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573686.748139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573686.748156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573686.748164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573686/real 1788573686] req@ffff920a52892680 x1875444569745280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573686.748172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569745280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573686.748177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573686.748179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573686.748181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573686.748184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573686.748195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573686.748200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573686/real 1788573686] req@ffff920a52890a80 x1875444569745408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573686.748208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569745408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573711 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573686.748213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573686.748215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573686.748217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573687.195759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573687.771698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573688.219791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573688.795765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573689.243812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573689.819756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573690.267797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573690.843690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573691.291741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573691.867693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573691.867696:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573691.867706:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573691.867711:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573691.867722:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10445 00000100:00080000:1.0:1788573691.867732:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573691.867741:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573691.867781:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569745536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573691.867809:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573691.867814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573691.867816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573691.867822:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10445 00000100:00080000:1.0:1788573691.867827:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573691.867832:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573691.867838:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569745536:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573691.867843:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573691.867861:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569745664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573691.867874:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573691.867878:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573691.867880:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573691.867886:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10445 00000100:00080000:1.0:1788573691.867890:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573691.867895:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573691.867901:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573691.867907:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573691.867916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569745792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573691.867929:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573691.867933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573691.867936:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573691.867940:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573691/real 1788573691] req@ffff920a52890000 x1875444569745536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573691.867944:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10445 00000100:00080000:1.0:1788573691.867952:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573691.867959:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573691.867964:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569745536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573691.867967:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573691.867981:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788573691.867987:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569745920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573691.867990:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573691.868000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569745536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573691.868010:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573691.868013:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573691.868015:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573691.868019:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10445 00000100:00080000:1.0:1788573691.868022:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573691.868026:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573691.868030:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573691.868030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573691.868047:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569746048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573691.868060:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573691.868073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573691/real 1788573691] req@ffff920a52893b80 x1875444569745664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573691.868083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569745664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573691.868090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573691.868092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573691.868095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573691.868099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573691.868112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573691.868118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573691/real 1788573691] req@ffff920a52890a80 x1875444569745792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573691.868130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569745792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573691.868136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573691.868137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573691.868139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573691.868143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573691.868154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573691.868160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573691/real 1788573691] req@ffff920a52892680 x1875444569745920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573691.868168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569745920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573691.868173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573691.868174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573691.868176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569745920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573691.868186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573691.868196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573691.868202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573691/real 1788573691] req@ffff920a52891500 x1875444569746048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573691.868209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569746048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573716 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573691.868218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573691.868220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573691.868222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573692.315845:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573692.891736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573693.339692:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573693.915695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573694.363743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573694.939716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573695.387718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573695.963716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573696.411811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573696.987670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573696.987679:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573696.987686:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573696.987691:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573696.987704:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10450 00000100:00080000:1.0:1788573696.987717:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573696.987726:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573696.987776:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569746176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573696.987804:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573696.987807:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573696.987809:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573696.987812:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10450 00000100:00080000:1.0:1788573696.987815:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573696.987818:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573696.987828:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573696.987838:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569746304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573696.987846:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573696.987849:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788573696.987849:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569746176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573696.987852:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573696.987859:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10450 00000100:00080000:1.0:1788573696.987861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573696.987863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573696.987866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573696.987875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569746432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573696.987882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573696.987885:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573696.987886:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573696.987889:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10450 00000100:00080000:1.0:1788573696.987892:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573696.987895:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573696.987899:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573696.987906:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569746560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573696.987912:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573696.987915:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573696.987916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573696.987919:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10450 00000100:00080000:1.0:1788573696.987921:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573696.987923:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573696.987926:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573696.987931:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573696.987939:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569746688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573696.987970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573696/real 1788573696] req@ffff920a52891f80 x1875444569746176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573696.987987:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569746176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573696.987996:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573696.987999:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573696.988002:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569746176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573696.988027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573696.988046:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573696.988057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573696/real 1788573696] req@ffff920a52892a00 x1875444569746304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573696.988066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569746304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573696.988072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573696.988075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573696.988077:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573696.988082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573696.988097:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573696.988103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573696/real 1788573696] req@ffff920a52893800 x1875444569746432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573696.988111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569746432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573696.988117:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573696.988119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573696.988121:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573696.988125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573696.988138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573696.988146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573696/real 1788573696] req@ffff920a52893480 x1875444569746560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573696.988153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569746560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573696.988159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573696.988161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573696.988163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573696.988169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573696.988190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573696.988198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573696/real 1788573696] req@ffff920a52890380 x1875444569746688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573696.988208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569746688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573721 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573696.988216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573696.988220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573696.988224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573697.435797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573698.011722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573698.459785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573699.035728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573699.483761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573700.059721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573700.507770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573701.083743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573701.531755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573702.107711:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573702.107712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573702.107721:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788573702.107726:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573702.107736:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10455 00000100:00080000:2.0:1788573702.107746:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573702.107754:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573702.107798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569746816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573702.107822:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573702.107824:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573702.107826:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573702.107829:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10455 00000100:00080000:2.0:1788573702.107831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573702.107834:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573702.107844:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573702.107852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569746816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573702.107857:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569746944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573702.107869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573702.107873:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573702.107874:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573702.107879:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10455 00000100:00080000:2.0:1788573702.107883:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573702.107887:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573702.107894:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573702.107908:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569747072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573702.107919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573702.107922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573702.107923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573702.107926:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10455 00000100:00080000:2.0:1788573702.107929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573702.107934:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788573702.107935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788573702.107941:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573702.107955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569747200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573702.107966:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573702.107970:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573702.107971:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573702.107975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff920a60ac4700 x1875444569746816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573702.107980:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10455 00000100:00080000:2.0:1788573702.107988:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573702.107994:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573702.108000:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573702.108000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569746816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573702.108012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788573702.108014:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569747328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573702.108020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573702.108029:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569746816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573702.108072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573702.108092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573702.108103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff920a60ac4a80 x1875444569746944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573702.108111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569746944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573702.108122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573702.108125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573702.108127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569746944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573702.108131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573702.108160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573702.108168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff920a60ac5500 x1875444569747072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573702.108177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569747072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573702.108183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573702.108186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573702.108188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573702.108192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573702.108206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573702.108213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff920a60ac6300 x1875444569747200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573702.108221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569747200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573702.108227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573702.108229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573702.108232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573702.108236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573702.108247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573702.108254:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573702/real 1788573702] req@ffff920a60ac7100 x1875444569747328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573702.108261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569747328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573702.108267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573702.108269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573702.108271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573702.555717:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573703.131737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573703.579786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573704.155710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573704.603786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573705.179725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573705.627791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573706.203765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573706.651840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573707.227684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573707.227723:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.227730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573707.227734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.227744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10460 00000100:00080000:3.0:1788573707.227755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.227763:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573707.227807:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569747456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.227842:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.227845:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573707.227847:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.227850:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10460 00000100:00080000:3.0:1788573707.227852:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.227856:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573707.227867:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573707.227877:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569747584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573707.227889:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569747456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573707.227891:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.227899:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573707.227901:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.227905:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10460 00000100:00080000:3.0:1788573707.227907:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.227910:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573707.227913:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573707.227922:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569747712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.227929:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.227932:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573707.227933:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.227936:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10460 00000100:00080000:3.0:1788573707.227937:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.227940:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573707.227943:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573707.227950:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569747840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573707.227955:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573707.227958:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573707.227959:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573707.227962:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10460 00000100:00080000:3.0:1788573707.227971:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573707.227973:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573707.227976:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573707.227985:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569747968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573707.227989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573707.228035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff920a529b0e00 x1875444569747456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.228056:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569747456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573707.228066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573707.228069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573707.228073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569747456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573707.228096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573707.228115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573707.228123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff920a529b1f80 x1875444569747584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.228136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569747584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573707.228145:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573707.228149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573707.228153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573707.228168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573707.228190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573707.228203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff920a529b0000 x1875444569747712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.228217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569747712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573707.228234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573707.228352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573707.228355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573707.228360:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573707.228379:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573707.228389:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff920a529b3100 x1875444569747840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.228402:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569747840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573707.228411:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573707.228415:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573707.228419:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573707.228426:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573707.228446:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573707.228454:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573707/real 1788573707] req@ffff920a529b2d80 x1875444569747968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573707.228462:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569747968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573707.228468:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573707.228470:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573707.228473:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569747968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573707.675759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573708.251692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573708.699854:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573709.275726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573709.723777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573710.299726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573710.747817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573711.323770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573711.771806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573712.347682:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.347698:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573712.347702:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.347712:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10465 00000100:00100000:2.0:1788573712.347716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573712.347723:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.347731:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573712.347764:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569748096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.347788:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.347791:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573712.347792:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.347796:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10465 00000100:00080000:3.0:1788573712.347798:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.347801:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573712.347810:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573712.347821:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569748224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573712.347829:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569748096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573712.347831:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.347837:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573712.347840:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.347844:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10465 00000100:00080000:3.0:1788573712.347846:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.347848:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573712.347851:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573712.347860:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569748352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.347870:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.347874:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573712.347876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.347880:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10465 00000100:00080000:3.0:1788573712.347883:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.347888:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573712.347894:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573712.347901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573712.347909:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569748480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573712.347921:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573712.347923:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573712.347925:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573712.347929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10465 00000100:00080000:3.0:1788573712.347933:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573712.347936:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573712.347941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573712.347942:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff920a529b0700 x1875444569748096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573712.347956:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569748608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573712.347966:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569748096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573712.347980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573712.347985:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573712.347991:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569748096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573712.348016:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573712.348039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573712.348052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff920a529b1c00 x1875444569748224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573712.348073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569748224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573712.348082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573712.348086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573712.348089:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573712.348095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573712.348130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573712.348142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff920a529b2300 x1875444569748352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573712.348155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569748352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573712.348163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573712.348166:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573712.348170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573712.348177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573712.348195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573712.348206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff920a529b3480 x1875444569748480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573712.348218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569748480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573712.348226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573712.348229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573712.348233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573712.348239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573712.348256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573712.348274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573712/real 1788573712] req@ffff920a529b2a00 x1875444569748608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573712.348291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569748608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573712.348306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573712.348312:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573712.348315:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573712.795727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573713.371766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573713.819758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573714.395738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573714.843808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573715.419670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573715.867767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573716.443700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573716.891768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573717.467701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573717.467701:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573717.467720:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573717.467725:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573717.467736:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10470 00000100:00080000:3.0:1788573717.467747:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573717.467756:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573717.467802:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569748736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573717.467825:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573717.467828:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573717.467829:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573717.467832:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10470 00000100:00080000:3.0:1788573717.467835:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573717.467838:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573717.467848:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573717.467858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569748736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573717.467863:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569748864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573717.467872:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573717.467875:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573717.467876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573717.467885:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10470 00000100:00080000:3.0:1788573717.467887:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573717.467890:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573717.467894:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573717.467903:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569748992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573717.467911:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573717.467913:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573717.467914:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573717.467916:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10470 00000100:00080000:3.0:1788573717.467925:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573717.467928:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573717.467931:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573717.467935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573717.467940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569749120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573717.467947:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573717.467950:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573717.467952:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573717.467955:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10470 00000100:00080000:3.0:1788573717.467959:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573717.467961:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573717.467966:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573717.467975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff920a529b2a00 x1875444569748736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573717.467976:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569749248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573717.468020:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569748736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573717.468030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573717.468034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573717.468036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569748736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573717.468060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573717.468086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573717.468096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff920a529b3480 x1875444569748864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573717.468105:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569748864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573717.468110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573717.468114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573717.468118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573717.468124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573717.468146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573717.468158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff920a529b2300 x1875444569748992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573717.468170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569748992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573717.468179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573717.468182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573717.468186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569748992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573717.468192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573717.468214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573717.468225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff920a529b1c00 x1875444569749120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573717.468244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569749120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573717.468252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573717.468255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573717.468258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573717.468265:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573717.468288:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573717.468300:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573717/real 1788573717] req@ffff920a529b0700 x1875444569749248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573717.468312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569749248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573717.468320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573717.468324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573717.468327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573717.915756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573718.491674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573718.939763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573719.515675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573719.963718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573720.539664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573720.987780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573721.563699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573722.011769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788573722.587710:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.587722:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788573722.587727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788573722.587727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.587742:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10475 00000100:00080000:2.0:1788573722.587757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.587766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573722.587829:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569749376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.587855:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.587858:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788573722.587860:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.587863:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10475 00000100:00080000:2.0:1788573722.587868:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.587873:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573722.587884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573722.587913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569749376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573722.587915:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569749504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.587931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.587934:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788573722.587936:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.587939:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10475 00000100:00080000:2.0:1788573722.587943:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.587946:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573722.587949:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573722.587960:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569749632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573722.587967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.587970:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788573722.587971:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.587973:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10475 00000100:00080000:2.0:1788573722.587976:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.587978:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573722.587982:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573722.587989:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569749760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573722.588002:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788573722.588003:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788573722.588007:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788573722.588009:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788573722.588012:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10475 00000100:00080000:2.0:1788573722.588014:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573722.588017:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788573722.588020:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573722.588028:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569749888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573722.588046:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff920a60ac5180 x1875444569749376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573722.588067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569749376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573722.588079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573722.588084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573722.588089:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569749376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573722.588118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573722.588145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573722.588159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff920a60ac6a00 x1875444569749504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573722.588168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569749504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573722.588174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573722.588176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573722.588179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573722.588183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573722.588197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573722.588203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff920a60ac7100 x1875444569749632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573722.588216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569749632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573722.588221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573722.588222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573722.588224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573722.588228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573722.588240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573722.588246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff920a60ac6300 x1875444569749760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573722.588253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569749760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573722.588258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573722.588259:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573722.588261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573722.588265:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573722.588275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573722.588280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573722/real 1788573722] req@ffff920a60ac5500 x1875444569749888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573722.588286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569749888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573722.588295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573722.588297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573722.588299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569749888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573723.035781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573723.611709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573724.059769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573724.635748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573725.083866:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573725.659791:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573726.107804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573726.683665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573727.131766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573727.707811:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573727.707817:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573727.707832:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573727.707839:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573727.707856:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10481 00000100:00080000:3.0:1788573727.707871:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573727.707882:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573727.707936:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569750016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573727.707983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573727.707992:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573727.707996:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573727.708007:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10481 00000100:00080000:3.0:1788573727.708014:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573727.708021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569750016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573727.708032:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573727.708052:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573727.708098:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569750144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573727.708121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573727.708134:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573727.708143:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573727.708148:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573727.708159:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10481 00000100:00080000:3.0:1788573727.708169:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573727.708174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573727/real 1788573727] req@ffff920a529b2d80 x1875444569750016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573727.708183:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573727.708203:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573727.708209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569750016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573727.708231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573727.708236:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569750272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573727.708240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573727.708262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569750016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573727.708270:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573727.708278:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573727.708283:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573727.708290:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10481 00000100:00080000:3.0:1788573727.708296:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573727.708303:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573727.708316:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573727.708320:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573727.708341:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569750400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788573727.708360:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573727.708363:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573727.708371:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573727.708376:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573727.708384:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573727/real 1788573727] req@ffff920a529b3100 x1875444569750144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573727.708386:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10481 00000100:00080000:3.0:1788573727.708398:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573727.708412:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573727.708429:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573727.708440:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569750144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573727.708452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573727.708456:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569750528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573727.708461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573727.708478:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573727.708514:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573727.708547:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573727.708562:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573727/real 1788573727] req@ffff920a529b0000 x1875444569750272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573727.708575:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569750272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573727.708634:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573727.708639:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573727.708643:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573727.708668:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573727.708701:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573727.708720:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573727/real 1788573727] req@ffff920a529b1f80 x1875444569750400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573727.708734:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569750400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573727.708750:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573727.708754:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573727.708758:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573727.708765:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573727.708785:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573727.708814:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573727/real 1788573727] req@ffff920a529b0e00 x1875444569750528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573727.708843:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569750528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573752 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573727.708853:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573727.708857:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573727.708861:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573728.155801:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573728.731728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573729.179785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573729.755691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573730.203793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573730.779687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573731.227840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573731.803740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573732.251773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573732.827681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573732.827681:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573732.827708:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573732.827717:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573732.827728:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10486 00000100:00080000:3.0:1788573732.827739:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573732.827748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573732.827784:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569750656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573732.827806:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573732.827809:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573732.827810:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573732.827813:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10486 00000100:00080000:3.0:1788573732.827817:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573732.827820:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573732.827830:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573732.827836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569750656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573732.827847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569750784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573732.827861:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573732.827865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573732.827868:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573732.827872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10486 00000100:00080000:3.0:1788573732.827877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573732.827881:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573732.827886:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788573732.827896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573732.827898:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569750912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573732.827907:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573732.827909:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573732.827911:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573732.827914:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10486 00000100:00080000:3.0:1788573732.827920:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573732.827922:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573732.827926:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573732.827933:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569751040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573732.827940:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573732/real 1788573732] req@ffff920a529b3b80 x1875444569750656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573732.827942:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573732.827949:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573732.827952:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573732.827959:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10486 00000100:00080000:3.0:1788573732.827963:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573732.827969:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573732.827974:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573732.827985:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569751168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573732.827996:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569750656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573732.828006:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573732.828010:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573732.828013:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569750656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573732.828033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573732.828054:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573732.828065:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573732/real 1788573732] req@ffff920a529b1500 x1875444569750784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573732.828076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569750784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573732.828083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573732.828085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573732.828087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573732.828092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573732.828109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573732.828118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573732/real 1788573732] req@ffff920a529b1180 x1875444569750912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573732.828126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569750912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573732.828137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573732.828140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573732.828142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569750912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573732.828154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573732.828170:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573732.828181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573732/real 1788573732] req@ffff920a529b0380 x1875444569751040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573732.828189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569751040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573732.828195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573732.828198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573732.828201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573732.828207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573732.828222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573732.828229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573732/real 1788573732] req@ffff920a529b0e00 x1875444569751168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573732.828237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569751168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573757 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573732.828243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573732.828245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573732.828248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573733.275758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573733.851701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573734.299750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573734.875708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573735.323768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573735.899685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573736.347762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573736.923700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573737.371750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573737.947690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573737.947719:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573737.947728:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573737.947732:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573737.947743:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10491 00000100:00080000:3.0:1788573737.947756:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573737.947766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573737.947821:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569751296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573737.947860:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573737.947864:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573737.947866:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573737.947870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10491 00000100:00080000:3.0:1788573737.947873:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573737.947877:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573737.947888:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573737.947903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569751424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573737.947912:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569751296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573737.947918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573737.947922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573737.947924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573737.947928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10491 00000100:00080000:3.0:1788573737.947933:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573737.947937:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573737.947941:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573737.947951:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569751552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573737.947960:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573737.947963:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573737.947966:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573737.947972:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10491 00000100:00080000:3.0:1788573737.947975:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573737.947981:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788573737.947992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573737.947996:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573737.948009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569751680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573737.948017:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573737.948020:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573737.948022:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573737.948028:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10491 00000100:00080000:3.0:1788573737.948032:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573737.948037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573737/real 1788573737] req@ffff920a529b1f80 x1875444569751296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573737.948037:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573737.948046:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573737.948061:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569751808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573737.948068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569751296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573737.948090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573737.948094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573737.948098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569751296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573737.948122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573737.948139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573737.948148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573737/real 1788573737] req@ffff920a529b0000 x1875444569751424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573737.948156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569751424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573737.948166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573737.948168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573737.948170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573737.948174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573737.948184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573737.948190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573737/real 1788573737] req@ffff920a529b3100 x1875444569751552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573737.948197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569751552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573737.948202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573737.948204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573737.948206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573737.948209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573737.948220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573737.948226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573737/real 1788573737] req@ffff920a529b2d80 x1875444569751680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573737.948233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569751680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573737.948237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573737.948239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573737.948242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573737.948246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573737.948256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573737.948261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573737/real 1788573737] req@ffff920a529b0700 x1875444569751808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573737.948268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569751808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573762 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573737.948274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573737.948276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573737.948277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569751808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573738.395766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573738.971667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573739.419796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573739.995681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573740.443770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573741.019669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573741.467713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573742.043738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573742.491779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573743.067664:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573743.067674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573743.067676:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573743.067681:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.067693:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10496 00000100:00080000:3.0:1788573743.067705:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.067714:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573743.067776:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569751936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.067809:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.067813:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.067815:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.067819:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10496 00000100:00080000:3.0:1788573743.067823:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.067827:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573743.067838:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573743.067843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569751936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573743.067856:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569752064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.067870:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.067873:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573743.067875:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.067880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10496 00000100:00080000:3.0:1788573743.067884:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.067889:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573743.067895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573743.067908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569752192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.067919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.067923:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573743.067925:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.067929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10496 00000100:00080000:3.0:1788573743.067933:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573743.067938:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573743.067944:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573743.067956:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573743.067958:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569752320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573743.067972:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573743.067976:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573743.067978:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573743.067982:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10496 00000100:00080000:3.0:1788573743.067987:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788573743.067994:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752192:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573743.067994:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573743.068003:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573743.068020:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569752448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573743.068040:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573743.068072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573743.515752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573744.091731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573744.539746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573745.115699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573745.563760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573746.139701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788573746.139900:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788573746.139945:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573746.139961:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573746.140015:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573746.140027:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573746.140036:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573746.140044:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788573746.140128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573746] req@ffff920a529b2a00 x1875444569752320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573746.140154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569752320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573746.140166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573746.140172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573746.140177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573746.140186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573746] req@ffff920a529b0700 x1875444569752448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573746.140199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569752448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573746.140209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573746.140212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573746.140216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752448:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573746.140224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573746] req@ffff920a529b2300 x1875444569752064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573746.140238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569752064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573746.140247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573746.140251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573746.140254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573746.140260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573746] req@ffff920a529b3480 x1875444569752192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573746.140271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569752192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573746.140278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573746.140282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573746.140286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573746.140293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573743/real 1788573746] req@ffff920a529b1c00 x1875444569751936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573746.140307:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569751936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573746.140315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573746.140319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573746.140322:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569751936:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788573746.587842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573747.163730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573747.611774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573748.187724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573748.187772:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573748.187778:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573748.187782:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573748.187792:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10501 00000100:00080000:1.0:1788573748.187804:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573748.187812:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573748.187854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569752576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573748.187881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573748.187884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573748.187886:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573748.187889:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10501 00000100:00080000:1.0:1788573748.187892:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573748.187895:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573748.187906:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573748.187917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569752704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573748.187924:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569752576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573748.187927:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573748.187932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573748.187934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573748.187938:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10501 00000100:00080000:1.0:1788573748.187940:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573748.187943:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573748.187948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573748.187961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569752832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573748.187970:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573748.187975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573748.187977:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573748.187981:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10501 00000100:00080000:1.0:1788573748.187985:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573748.187997:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573748.188004:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788573748.188010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573748.188019:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569752960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573748.188032:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573748.188036:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573748.188039:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573748.188043:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10501 00000100:00080000:1.0:1788573748.188049:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573748.188054:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573748.188058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff920a52890e00 x1875444569752576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573748.188061:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573748.188079:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569753088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573748.188090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569752576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.188104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.188110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573748.188116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569752576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573748.188149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573748.188178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573748.188194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff920a52891180 x1875444569752704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573748.188212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569752704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.188221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.188225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573748.188229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573748.188236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573748.188260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573748.188272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff920a52892300 x1875444569752832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573748.188286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569752832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.188308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.188312:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573748.188315:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573748.188321:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573748.188343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573748.188354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff920a52892d80 x1875444569752960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573748.188364:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569752960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.188377:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.188397:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573748.188401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569752960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573748.188408:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573748.188428:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573748.188439:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573748/real 1788573748] req@ffff920a52891880 x1875444569753088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573748.188451:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569753088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573748.188460:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573748.188463:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573748.188466:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573748.635844:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573749.211741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573749.659780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573750.235716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573750.683762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573751.259666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573751.707751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573752.283701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573752.731733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573753.307669:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573753.307671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573753.307681:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573753.307686:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573753.307694:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10506 00000100:00080000:1.0:1788573753.307704:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573753.307710:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573753.307749:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569753216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573753.307772:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573753.307775:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573753.307777:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573753.307780:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10506 00000100:00080000:1.0:1788573753.307783:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573753.307786:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573753.307795:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573753.307804:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569753344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573753.307814:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573753.307814:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569753216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573753.307822:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573753.307825:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573753.307829:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10506 00000100:00080000:1.0:1788573753.307831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573753.307833:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573753.307836:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573753.307845:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569753472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573753.307852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573753.307855:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573753.307856:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573753.307859:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10506 00000100:00080000:1.0:1788573753.307861:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573753.307863:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573753.307866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573753.307873:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569753600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573753.307879:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573753.307892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573753.307894:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573753.307896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573753.307899:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10506 00000100:00080000:1.0:1788573753.307903:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573753.307908:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573753.307913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573753.307920:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff920a52893100 x1875444569753216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573753.307928:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569753728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573753.307938:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569753216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.307949:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.307952:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573753.307957:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569753216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573753.307976:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573753.307992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573753.308000:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff920a52890700 x1875444569753344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.308008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569753344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.308013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.308015:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573753.308017:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573753.308021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573753.308031:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573753.308037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff920a52891880 x1875444569753472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.308044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569753472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.308049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.308050:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573753.308052:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573753.308056:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573753.308066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573753.308071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff920a52892d80 x1875444569753600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.308078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569753600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.308083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.308085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573753.308087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573753.308090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573753.308100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573753.308105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573753/real 1788573753] req@ffff920a52892300 x1875444569753728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573753.308112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569753728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573753.308122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573753.308124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573753.308126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573753.755775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573754.331711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573754.779753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573755.355677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573755.803739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573756.379695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573756.827726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573757.403699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573757.851838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573758.427684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573758.427694:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573758.427700:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573758.427705:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573758.427716:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10511 00000100:00080000:0.0:1788573758.427726:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573758.427734:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573758.427786:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569753856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573758.427810:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573758.427812:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573758.427814:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573758.427817:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10511 00000100:00080000:0.0:1788573758.427819:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573758.427821:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573758.427831:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573758.427839:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569753984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573758.427851:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573758.427856:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573758.427857:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573758.427861:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10511 00000100:00080000:0.0:1788573758.427864:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573758.427867:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573758.427870:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573758.427877:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569754112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573758.427888:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573758.427891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573758.427893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573758.427896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569753856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573758.427898:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10511 00000100:00080000:0.0:1788573758.427902:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573758.427907:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573758.427911:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573758.427920:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569754240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573758.427925:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573758.427928:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573758.427929:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573758.427932:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10511 00000100:00080000:0.0:1788573758.427934:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573758.427936:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573758.427939:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573758.427947:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569754368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573758.427986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573758.428024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff920a48353b80 x1875444569753856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573758.428050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569753856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573758.428063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573758.428068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573758.428073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569753856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573758.428081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573758.428102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573758.428112:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff920a48353800 x1875444569753984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573758.428121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569753984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573758.428127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573758.428129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573758.428132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569753984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573758.428153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573758.428164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573758.428170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff920a48351c00 x1875444569754112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573758.428204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569754112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573758.428216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573758.428218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573758.428221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573758.428225:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573758.428240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573758.428248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff920a48351500 x1875444569754240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573758.428255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569754240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573758.428260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573758.428262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573758.428264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573758.428267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573758.428278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573758.428283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573758/real 1788573758] req@ffff920a48350a80 x1875444569754368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573758.428291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569754368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573758.428295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573758.428297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573758.428300:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573758.875782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573759.451702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573759.899772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573760.475716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573760.923742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573761.499697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573761.947753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788573762.523653:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573762.971784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573763.547681:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573763.547692:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788573763.547693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573763.547698:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573763.547711:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10516 00000100:00080000:3.0:1788573763.547722:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573763.547730:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573763.547768:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569754496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573763.547792:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573763.547795:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573763.547797:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573763.547800:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10516 00000100:00080000:3.0:1788573763.547804:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573763.547808:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573763.547817:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573763.547822:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569754496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573763.547829:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569754624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573763.547838:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573763.547841:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573763.547843:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573763.547845:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10516 00000100:00080000:3.0:1788573763.547848:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573763.547850:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573763.547854:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573763.547862:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569754752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573763.547868:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573763.547871:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573763.547872:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573763.547875:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10516 00000100:00080000:3.0:1788573763.547877:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573763.547879:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573763.547883:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573763.547887:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573763.547893:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569754880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573763.547900:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573763.547903:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573763.547904:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573763.547907:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10516 00000100:00080000:3.0:1788573763.547910:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573763.547912:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573763.547916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573763.547923:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff920a529b1c00 x1875444569754496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573763.547925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569755008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573763.547961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569754496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573763.547979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573763.547982:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573763.547985:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569754496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573763.548002:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573763.548019:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573763.548028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff920a529b3480 x1875444569754624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573763.548036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569754624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573763.548041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573763.548043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573763.548045:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573763.548049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573763.548059:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573763.548064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff920a529b2300 x1875444569754752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573763.548076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569754752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573763.548104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573763.548107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573763.548109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573763.548113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573763.548134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573763.548142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff920a529b0700 x1875444569754880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573763.548150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569754880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573763.548155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573763.548157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573763.548159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569754880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573763.548162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573763.548177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573763.548189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573763/real 1788573763] req@ffff920a529b2a00 x1875444569755008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573763.548201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569755008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573763.548216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573763.548219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573763.548222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573763.995784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573764.571722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573765.019828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573765.595702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573766.043755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573766.619740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573767.067781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573767.643700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573768.091747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573768.667699:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573768.667708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573768.667712:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573768.667721:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10522 00000100:00080000:3.0:1788573768.667740:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573768.667746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573768.667749:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573768.667786:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569755136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573768.667803:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573768.667806:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573768.667808:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573768.667812:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10522 00000100:00080000:3.0:1788573768.667814:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573768.667819:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573768.667830:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573768.667841:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569755136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573768.667864:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569755264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573768.667878:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573768.667881:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573768.667883:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573768.667887:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10522 00000100:00080000:3.0:1788573768.667892:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573768.667896:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573768.667901:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573768.667923:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569755392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573768.667934:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573768.667941:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788573768.667942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573768.667945:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573768.667952:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10522 00000100:00080000:3.0:1788573768.667957:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573768.667962:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573768.667969:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573768.667995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573768/real 1788573768] req@ffff920a529b2d80 x1875444569755136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573768.668027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569755136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573768.668035:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573768.668039:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573768.668041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569755136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573768.668061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573768.668086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573768.668100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573768/real 1788573768] req@ffff920a529b3100 x1875444569755264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573768.668121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569755264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573768.668130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573768.668134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573768.668136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573768.668141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573768.668158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573768.668166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573768/real 1788573768] req@ffff920a529b0000 x1875444569755392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573768.668174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569755392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573768.668179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573768.668181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573768.668184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573768.668335:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569755520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573768.668351:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573768.668354:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573768.668356:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573768.668386:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755520:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788573768.668394:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10522 00000100:00080000:3.0:1788573768.668399:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573768.668403:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573768.668408:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573768.668419:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573768.668421:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569755648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573768.668438:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573768/real 1788573768] req@ffff920a529b1f80 x1875444569755520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573768.668451:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569755520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573768.668458:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573768.668461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573768.668463:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573768.668476:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573768.668497:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573768.668509:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573768/real 1788573768] req@ffff920a529b0e00 x1875444569755648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573768.668522:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569755648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573793 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573768.668531:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573768.668535:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573768.668539:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573769.115803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573769.691709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573770.139778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573770.715739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573771.163758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573771.739737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573772.187769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573772.763695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573773.211730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573773.787709:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573773.787723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573773.787723:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573773.787729:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573773.787747:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10527 00000100:00080000:3.0:1788573773.787760:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573773.787770:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573773.787816:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569755776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573773.787857:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573773.787861:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573773.787863:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573773.787868:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10527 00000100:00080000:3.0:1788573773.787872:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573773.787889:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573773.787901:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573773.787928:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569755776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573773.787930:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569755904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573773.787945:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573773.787949:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573773.787951:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573773.787956:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10527 00000100:00080000:3.0:1788573773.787961:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573773.787973:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573773.787980:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573773.787993:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569756032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573773.788005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573773.788006:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573773.788019:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573773.788021:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573773.788028:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10527 00000100:00080000:3.0:1788573773.788033:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573773.788038:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573773.788044:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573773.788047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573773/real 1788573773] req@ffff920a529b0380 x1875444569755776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573773.788061:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569756160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573773.788073:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573773.788075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569755776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573773.788080:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573773.788084:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573773.788093:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10527 00000100:00080000:2.0:1788573773.788094:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573773.788101:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788573773.788102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573773.788108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569755776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573773.788112:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573773.788120:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573773.788135:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569756288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573773.788137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573773.788173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573773.788183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573773/real 1788573773] req@ffff920a529b1180 x1875444569755904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573773.788192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569755904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573773.788202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573773.788206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573773.788208:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569755904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573773.788212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573773.788225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573773.788232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573773/real 1788573773] req@ffff920a529b1500 x1875444569756032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573773.788239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569756032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573773.788245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573773.788246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573773.788248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573773.788252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573773.788263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573773.788269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573773/real 1788573773] req@ffff920a529b3b80 x1875444569756160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573773.788276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569756160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573773.788282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573773.788283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573773.788285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573773.788295:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573773.788306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573773.788312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573773/real 1788573773] req@ffff920a529b0e00 x1875444569756288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573773.788319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569756288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573798 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573773.788324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573773.788325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573773.788327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573774.235784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573774.811685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573775.259806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573775.835721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573776.283801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573776.859681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573777.307770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573777.883678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573778.331731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573778.907732:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573778.907733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573778.907747:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573778.907752:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573778.907763:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10532 00000100:00080000:1.0:1788573778.907791:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573778.907801:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573778.907843:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569756416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573778.907877:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573778.907881:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573778.907883:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573778.907888:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10532 00000100:00080000:1.0:1788573778.907893:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573778.907898:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573778.907910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573778.907926:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569756416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573778.907927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569756544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573778.907943:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573778.907948:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573778.907950:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573778.907955:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10532 00000100:00080000:1.0:1788573778.907960:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573778.907965:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573778.907971:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788573778.907986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573778.907988:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569756672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573778.908002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573778.908007:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573778.908009:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573778.908014:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10532 00000100:00080000:1.0:1788573778.908019:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573778.908023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573778/real 1788573778] req@ffff920a52891180 x1875444569756416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573778.908028:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573778.908039:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573778.908046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569756416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573778.908056:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569756800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573778.908058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573778.908067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573778.908074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569756416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573778.908090:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573778.908094:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573778.908096:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573778.908101:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10532 00000100:00100000:2.0:1788573778.908104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573778.908109:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573778.908117:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573778.908124:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573778.908128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573778.908145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573778/real 1788573778] req@ffff920a52890e00 x1875444569756544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573778.908146:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569756928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573778.908170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569756544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573778.908179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573778.908183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573778.908185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573778.908190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573778.908204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573778.908211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573778/real 1788573778] req@ffff920a52890380 x1875444569756672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573778.908218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569756672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573778.908233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573778.908236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573778.908238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573778.908242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573778.908255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573778.908262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573778/real 1788573778] req@ffff920a52893480 x1875444569756800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573778.908269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569756800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573778.908275:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573778.908276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573778.908278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573778.908288:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573778.908299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573778.908305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573778/real 1788573778] req@ffff920a52893800 x1875444569756928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573778.908312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569756928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573803 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573778.908323:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573778.908325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573778.908327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569756928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573779.355823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573779.931680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573780.379800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573780.955700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573781.403758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573781.979657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573782.427724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573783.003827:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573783.451812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573784.027727:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573784.027739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573784.027740:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573784.027746:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573784.027763:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10537 00000100:00080000:3.0:1788573784.027778:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573784.027789:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573784.027841:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569757056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573784.027871:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573784.027874:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573784.027876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573784.027880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10537 00000100:00080000:3.0:1788573784.027884:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573784.027888:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573784.027899:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573784.027908:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569757056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573784.027911:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569757184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573784.027924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573784.027927:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573784.027928:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573784.027931:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10537 00000100:00080000:3.0:1788573784.027934:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573784.027936:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573784.027939:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573784.027949:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569757312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573784.027956:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573784.027960:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573784.027961:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573784.027964:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10537 00000100:00080000:3.0:1788573784.027977:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573784.027980:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573784.027984:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573784.027989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573784.027994:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569757440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573784.028003:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573784.028006:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573784.028008:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573784.028010:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10537 00000100:00080000:3.0:1788573784.028015:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573784.028017:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573784.028021:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573784.028029:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569757568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573784.028044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff920a529b1f80 x1875444569757056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.028072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569757056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.028081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.028085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573784.028088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569757056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573784.028113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573784.028132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573784.028141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff920a529b0000 x1875444569757184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.028150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569757184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.028156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.028158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573784.028160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573784.028165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573784.028176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573784.028182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff920a529b3100 x1875444569757312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.028189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569757312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.028200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.028202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573784.028205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573784.028208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573784.028220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573784.028226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff920a529b2d80 x1875444569757440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.028233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569757440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.028239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.028241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573784.028243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573784.028246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573784.028257:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573784.028263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573784/real 1788573784] req@ffff920a529b2a00 x1875444569757568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573784.028270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569757568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573809 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573784.028275:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573784.028277:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573784.028280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573784.475737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573785.051691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573785.499814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573786.075721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573786.523797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573787.099726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573787.547787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573788.123729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573788.571752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573789.147699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573789.147711:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573789.147719:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573789.147723:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573789.147733:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10542 00000100:00080000:0.0:1788573789.147741:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573789.147748:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573789.147781:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569757696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573789.147805:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573789.147814:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573789.147815:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573789.147820:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10542 00000100:00080000:0.0:1788573789.147823:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573789.147826:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788573789.147835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569757696:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573789.147836:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573789.147849:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569757824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573789.147857:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573789.147860:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573789.147862:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573789.147865:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10542 00000100:00080000:0.0:1788573789.147868:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573789.147870:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573789.147873:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573789.147881:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569757952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573789.147893:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788573789.147896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573789.147897:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573789.147901:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573789.147904:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10542 00000100:00080000:0.0:1788573789.147907:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573789.147908:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573789.147912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573789.147921:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569758080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573789.147928:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573789.147930:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573789.147932:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573789.147936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff920a48350700 x1875444569757696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573789.147938:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10542 00000100:00080000:0.0:1788573789.147942:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573789.147948:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573789.147953:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573789.147959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569757696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573789.147963:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569758208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573789.147980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.147986:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573789.147989:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569757696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573789.148008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573789.148025:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573789.148033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff920a48350380 x1875444569757824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.148042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569757824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.148048:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.148050:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573789.148052:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573789.148056:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573789.148068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573789.148074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff920a48353480 x1875444569757952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.148082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569757952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.148087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.148089:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573789.148090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569757952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573789.148094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573789.148104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573789.148110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff920a48350a80 x1875444569758080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.148117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569758080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.148122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.148124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573789.148126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573789.148129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573789.148138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573789.148144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573789/real 1788573789] req@ffff920a48351500 x1875444569758208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573789.148155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569758208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573814 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573789.148160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573789.148162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573789.148164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573789.595737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573790.171673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573790.619742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573791.195703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573791.643778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573792.219710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573792.667755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573793.243703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573793.691742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573794.267688:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573794.267700:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573794.267704:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573794.267715:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10547 00000100:00100000:2.0:1788573794.267722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573794.267725:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573794.267735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573794.267768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569758336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573794.267790:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573794.267793:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573794.267795:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573794.267798:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10547 00000100:00080000:1.0:1788573794.267801:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573794.267804:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573794.267813:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573794.267819:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569758336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573794.267823:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569758464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573794.267833:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573794.267836:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573794.267837:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573794.267840:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10547 00000100:00080000:1.0:1788573794.267842:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573794.267844:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573794.267847:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573794.267856:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569758592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573794.267862:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573794.267864:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573794.267865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573794.267868:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10547 00000100:00080000:1.0:1788573794.267870:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573794.267872:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573794.267875:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573794.267880:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573794.267884:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569758720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573794.267892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573794.267894:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573794.267896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573794.267899:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10547 00000100:00080000:1.0:1788573794.267901:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573794.267903:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573794.267906:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573794.267917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569758848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573794.267917:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573794] req@ffff920a52892a00 x1875444569758336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573794.267962:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569758336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573794.267974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573794.267978:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573794.267983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569758336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573794.268006:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573794.268030:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573794.268044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573794] req@ffff920a52891f80 x1875444569758464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573794.268066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569758464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573794.268076:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573794.268080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573794.268083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573794.268088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573794.268105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573794.268113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573794] req@ffff920a52891500 x1875444569758592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573794.268122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569758592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573794.268128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573794.268131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573794.268133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573794.268136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573794.268147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573794.268153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573794] req@ffff920a52892680 x1875444569758720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573794.268161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569758720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573794.268171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573794.268173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573794.268175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573794.268179:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573794.268188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573794.268194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573794/real 1788573794] req@ffff920a52890a80 x1875444569758848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573794.268200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569758848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573819 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573794.268205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573794.268207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573794.268209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569758848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573794.715743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573795.291697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573795.739811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573796.315726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573796.763759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573797.339699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573797.787713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573798.363714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573798.811867:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573799.387693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573799.387728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573799.387738:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573799.387742:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573799.387753:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10552 00000100:00080000:1.0:1788573799.387762:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573799.387769:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573799.387810:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569758976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573799.387836:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573799.387840:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573799.387842:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573799.387846:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10552 00000100:00080000:1.0:1788573799.387848:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573799.387851:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573799.387861:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573799.387872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569759104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573799.387883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573799.387886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573799.387891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573799.387892:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569758976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573799.387900:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10552 00000100:00080000:1.0:1788573799.387904:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573799.387907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573799.387911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573799.387921:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569759232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573799.387930:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573799.387933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573799.387935:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573799.387938:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10552 00000100:00080000:1.0:1788573799.387940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573799.387942:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573799.387947:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573799.387956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569759360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573799.387962:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573799.387965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573799.387966:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573799.387969:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10552 00000100:00080000:1.0:1788573799.387974:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573799.387977:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573799.387980:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573799.387993:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569759488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573799.387993:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573799.388050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff920a52893b80 x1875444569758976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573799.388082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569758976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.388093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573799.388096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573799.388099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569758976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573799.388126:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573799.388155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573799.388172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff920a52890000 x1875444569759104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573799.388188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569759104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.388199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573799.388203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573799.388207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573799.388215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573799.388241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573799.388253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff920a52890a80 x1875444569759232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573799.388265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569759232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.388282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573799.388287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573799.388291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573799.388299:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573799.388324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573799.388333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff920a52892680 x1875444569759360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573799.388342:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569759360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.388348:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573799.388351:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573799.388353:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573799.388357:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573799.388370:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573799.388377:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573799/real 1788573799] req@ffff920a52891500 x1875444569759488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573799.388385:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569759488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573824 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573799.388390:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573799.388392:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573799.388395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573799.835763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573800.411687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573800.859765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573801.435658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573801.883775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573802.459722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573802.907793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573803.483691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573803.931758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573804.507739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573804.507784:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573804.507795:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573804.507800:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573804.507810:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10557 00000100:00080000:3.0:1788573804.507823:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573804.507834:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573804.507883:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569759616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573804.507917:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573804.507922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573804.507924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573804.507931:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10557 00000100:00080000:3.0:1788573804.507938:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573804.507944:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573804.507956:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573804.507964:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569759616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573804.507978:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569759744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573804.507994:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573804.508000:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573804.508003:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573804.508009:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10557 00000100:00080000:3.0:1788573804.508015:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573804.508021:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573804.508028:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573804.508051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573804.508091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff920a529b0700 x1875444569759616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.508110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569759616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573804.508121:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569759872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573804.508123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.508133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573804.508147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569759616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573804.508165:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573804.508170:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573804.508172:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573804.508185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759744:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573804.508185:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10557 00000800:00000100:1.0:1788573804.508205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573804.508214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff920a529b2300 x1875444569759744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.508224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569759744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.508230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.508233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573804.508235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573804.508239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573804.508251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573804.508258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff920a529b3480 x1875444569759872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.508267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569759872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.508278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.508281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573804.508283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569759872:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573804.508358:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573804.508366:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573804.508421:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573804.508445:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569760000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573804.508469:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573804.508474:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573804.508477:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573804.508485:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10557 00000100:00100000:1.0:1788573804.508488:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760000:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788573804.508496:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573804.508505:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573804.508512:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573804.508518:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573804.508534:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff920a529b1c00 x1875444569760000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573804.508545:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569760128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573804.508551:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569760000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.508612:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.508620:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573804.508632:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573804.508652:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573804.508678:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573804.508686:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573804/real 1788573804] req@ffff920a529b3480 x1875444569760128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573804.508694:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569760128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573829 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573804.508700:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573804.508702:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573804.508705:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573804.955784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573805.531714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573805.979778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573806.555639:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573807.003731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573807.579682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573808.027783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573808.603726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573809.051793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573809.627680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573809.627680:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573809.627694:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573809.627700:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573809.627713:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10563 00000100:00080000:3.0:1788573809.627736:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573809.627745:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573809.627794:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569760256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573809.627829:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573809.627834:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573809.627836:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573809.627840:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10563 00000100:00080000:3.0:1788573809.627843:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573809.627847:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573809.627856:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573809.627871:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569760384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573809.627879:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573809.627882:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573809.627884:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573809.627892:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10563 00000100:00100000:1.0:1788573809.627895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569760256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573809.627897:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573809.627907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573809.627922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573809.627937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569760512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573809.627946:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573809.627950:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573809.627951:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573809.627954:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10563 00000100:00080000:3.0:1788573809.627958:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573809.627961:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573809.627966:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573809.627973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569760640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573809.627981:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573809.627983:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573809.627985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573809.627987:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10563 00000100:00080000:3.0:1788573809.627989:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573809.627991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573809.627994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573809.628001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569760768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573809.628031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573809.628080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573809.628109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573809.628138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573810.075785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573810.651735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573811.099820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573811.675715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573812.123747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573812.699720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788573812.700076:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788573812.700121:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573812.700137:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573812.700194:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573812.700206:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573812.700213:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573812.700222:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573812.701426:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573809/real 1788573812] req@ffff920a529b2a00 x1875444569760512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573812.701457:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569760512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573812.701470:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573812.701476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573812.701480:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573812.701489:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573809/real 1788573812] req@ffff920a529b2d80 x1875444569760640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573812.701508:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569760640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573812.701516:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573812.701520:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573812.701523:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573812.701530:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573809/real 1788573812] req@ffff920a529b3100 x1875444569760768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573812.701541:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569760768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573812.701549:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573812.701553:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573812.701556:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760768:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573812.701564:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573809/real 1788573812] req@ffff920a529b2300 x1875444569760256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573812.701577:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569760256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573812.701609:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573812.701615:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573812.701622:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569760256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573812.701635:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573809/real 1788573812] req@ffff920a529b0700 x1875444569760384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573812.701651:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569760384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573834 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573812.701659:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573812.701663:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573812.701666:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569760384:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788573813.147795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573813.723679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573814.171805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573814.747736:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573814.747748:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788573814.747750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573814.747753:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573814.747774:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10568 00000100:00080000:0.0:1788573814.747787:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573814.747797:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573814.747845:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569760896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573814.747875:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573814.747879:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573814.747881:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573814.747885:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10568 00000100:00080000:0.0:1788573814.747889:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573814.747895:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573814.747906:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573814.747920:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569761024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573814.747934:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569760896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573814.747938:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573814.747948:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573814.747950:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573814.747956:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10568 00000100:00080000:0.0:1788573814.747961:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573814.747966:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573814.747972:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573814.747992:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569761152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573814.748003:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573814.748008:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573814.748010:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573814.748015:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10568 00000100:00080000:0.0:1788573814.748034:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573814.748039:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573814.748049:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573814.748057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573814.748069:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569761280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573814.748086:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573814.748089:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573814.748092:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573814.748096:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10568 00000100:00080000:0.0:1788573814.748102:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573814.748109:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573814.748114:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573814.748126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573814/real 1788573814] req@ffff920a48351c00 x1875444569760896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573814.748133:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569761408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573814.748207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569760896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573814.748223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573814.748229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573814.748233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569760896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573814.748267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573814.748298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573814.748314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573814/real 1788573814] req@ffff920a48353800 x1875444569761024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573814.748337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569761024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573814.748347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573814.748351:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573814.748355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573814.748363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573814.748387:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573814.748400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573814/real 1788573814] req@ffff920a48353b80 x1875444569761152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573814.748414:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569761152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573814.748423:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573814.748427:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573814.748431:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573814.748439:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573814.748463:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573814.748476:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573814/real 1788573814] req@ffff920a48351500 x1875444569761280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573814.748488:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569761280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573814.748502:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573814.748506:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573814.748509:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573814.748516:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573814.748538:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573814.748550:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573814/real 1788573814] req@ffff920a48350a80 x1875444569761408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573814.748562:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569761408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573839 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573814.748571:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573814.748574:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573814.748578:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573815.195749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573815.771724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573816.219854:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573816.795669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573817.243738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573817.819713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573818.267751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573818.843688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573819.291807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573819.867699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573819.867726:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573819.867734:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573819.867739:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573819.867749:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10573 00000100:00080000:3.0:1788573819.867761:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573819.867770:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573819.867803:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569761536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573819.867825:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573819.867828:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573819.867830:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573819.867833:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10573 00000100:00080000:3.0:1788573819.867836:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573819.867839:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573819.867847:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573819.867857:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569761664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573819.867865:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573819.867869:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569761536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573819.867869:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573819.867872:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573819.867880:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10573 00000100:00080000:3.0:1788573819.867882:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573819.867885:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573819.867888:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573819.867898:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569761792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573819.867904:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573819.867908:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573819.867910:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573819.867912:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10573 00000100:00080000:3.0:1788573819.867915:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573819.867917:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573819.867921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573819.867929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569761920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573819.867935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573819.867941:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573819.867945:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573819.867946:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573819.867950:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10573 00000100:00080000:3.0:1788573819.867960:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573819.867963:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573819.867967:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573819.867972:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573819/real 1788573819] req@ffff920a529b0000 x1875444569761536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573819.867976:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569762048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573819.868012:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569761536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573819.868021:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573819.868024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573819.868027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569761536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573819.868045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573819.868089:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573819.868104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573819/real 1788573819] req@ffff920a529b1f80 x1875444569761664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573819.868116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569761664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573819.868124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573819.868127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573819.868130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573819.868137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573819.868156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573819.868179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573819/real 1788573819] req@ffff920a529b0e00 x1875444569761792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573819.868194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569761792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573819.868210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573819.868214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573819.868216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573819.868221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573819.868238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573819.868247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573819/real 1788573819] req@ffff920a529b3b80 x1875444569761920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573819.868254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569761920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573819.868259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573819.868261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573819.868263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569761920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573819.868267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573819.868277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573819.868283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573819/real 1788573819] req@ffff920a529b1500 x1875444569762048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573819.868290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569762048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573844 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573819.868295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573819.868297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573819.868299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573820.315758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573820.891685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573821.339781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573821.915666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573822.363767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573822.939694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573823.387734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573823.963712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573824.411708:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573824.987668:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573824.987672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573824.987679:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573824.987683:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573824.987692:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10578 00000100:00080000:3.0:1788573824.987702:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573824.987708:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573824.987741:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569762176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573824.987768:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573824.987771:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573824.987773:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573824.987776:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10578 00000100:00080000:3.0:1788573824.987778:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573824.987782:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573824.987790:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573824.987799:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569762176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573824.987802:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569762304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573824.987813:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573824.987815:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573824.987816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573824.987819:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10578 00000100:00080000:3.0:1788573824.987822:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573824.987824:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573824.987828:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573824.987837:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569762432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573824.987844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573824.987846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573824.987847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573824.987850:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10578 00000100:00080000:3.0:1788573824.987853:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573824.987856:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573824.987859:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573824.987864:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573824.987870:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569762560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573824.987878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573824.987881:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573824.987882:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573824.987885:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10578 00000100:00080000:3.0:1788573824.987889:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573824.987893:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573824.987899:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573824.987900:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573824/real 1788573824] req@ffff920a529b1180 x1875444569762176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573824.987915:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569762688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573824.987926:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569762176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573824.987940:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573824.987946:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573824.987951:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569762176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573824.987998:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573824.988019:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573824.988029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573824/real 1788573824] req@ffff920a529b0380 x1875444569762304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573824.988037:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569762304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573824.988043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573824.988046:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573824.988048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573824.988052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573824.988064:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573824.988070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573824/real 1788573824] req@ffff920a529b1500 x1875444569762432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573824.988077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569762432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573824.988082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573824.988084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573824.988086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573824.988089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573824.988100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573824.988105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573824/real 1788573824] req@ffff920a529b3b80 x1875444569762560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573824.988113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569762560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573824.988118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573824.988119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573824.988121:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573824.988124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573824.988135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573824.988140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573824/real 1788573824] req@ffff920a529b0e00 x1875444569762688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573824.988147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569762688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573849 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573824.988151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573824.988154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573824.988156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573825.435763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573826.011738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573826.459785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573827.035697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573827.483756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573828.059739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573828.507767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573829.083686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573829.531759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573830.107731:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573830.107736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573830.107744:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573830.107749:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.107759:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10583 00000100:00080000:3.0:1788573830.107770:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.107777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573830.107811:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569762816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.107833:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.107837:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573830.107838:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.107841:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10583 00000100:00080000:3.0:1788573830.107845:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.107851:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573830.107862:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573830.107863:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569762816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573830.107913:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569762944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.107926:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.107931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573830.107933:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.107940:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10583 00000800:00000100:1.0:1788573830.107942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573830.107949:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.107956:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573830.107963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573830.107979:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569763072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573830.107986:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff920a529b1f80 x1875444569762816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573830.107992:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.108000:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573830.108001:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.108006:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10583 00000100:00100000:1.0:1788573830.108009:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569762816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573830.108011:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573830.108020:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573830.108024:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788573830.108027:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573830.108030:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573830.108035:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569762816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573830.108040:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569763200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573830.108048:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573830.108051:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573830.108053:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573830.108056:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10583 00000100:00080000:3.0:1788573830.108058:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573830.108060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762944:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573830.108063:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573830.108069:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573830.108078:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569763328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573830.108087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573830.108100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff920a529b0000 x1875444569762944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.108109:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569762944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.108115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.108118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573830.108120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569762944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573830.108124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573830.108135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573830.108141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff920a529b0700 x1875444569763072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.108149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569763072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.108154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.108156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573830.108158:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573830.108161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573830.108172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573830.108178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff920a529b2300 x1875444569763200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.108186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569763200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.108190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.108192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573830.108194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573830.108207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573830.108226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573830.108237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573830/real 1788573830] req@ffff920a529b3100 x1875444569763328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573830.108248:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569763328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573855 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573830.108255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573830.108258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573830.108261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573830.555762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573831.131665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573831.579725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573832.155753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573832.603765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573833.179691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573833.627755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573834.203731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573834.651745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573835.227688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573835.227722:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573835.227730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573835.227734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573835.227744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10588 00000100:00080000:0.0:1788573835.227755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573835.227768:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573835.227827:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569763456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573835.227849:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573835.227852:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573835.227854:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573835.227858:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10588 00000100:00080000:0.0:1788573835.227867:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573835.227871:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573835.227877:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569763456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573835.227881:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573835.227894:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569763584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573835.227902:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573835.227905:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573835.227907:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573835.227910:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10588 00000100:00080000:0.0:1788573835.227922:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573835.227925:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573835.227928:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573835.227935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573835.227938:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569763712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573835.227950:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573835.227954:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573835.227956:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573835.227960:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10588 00000100:00080000:0.0:1788573835.227965:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573835.227971:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573835.227976:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff920a48353480 x1875444569763456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788573835.227978:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573835.228001:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569763840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573835.228002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569763456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573835.228033:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573835.228034:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573835.228040:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573835.228040:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788573835.228044:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573835.228047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569763456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573835.228051:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10588 00000100:00080000:0.0:1788573835.228056:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573835.228059:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573835.228063:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573835.228071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573835.228075:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569763968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573835.228094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573835.228106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff920a48350380 x1875444569763584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.228115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569763584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.228120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.228123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573835.228125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573835.228128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573835.228139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573835.228145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff920a48350700 x1875444569763712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.228157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569763712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.228163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.228165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573835.228166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573835.228170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573835.228180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573835.228186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff920a48350a80 x1875444569763840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.228194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569763840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.228200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.228201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573835.228203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573835.228212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573835.228222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573835.228228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573835/real 1788573835] req@ffff920a48351500 x1875444569763968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573835.228235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569763968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573860 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573835.228245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573835.228247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573835.228249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569763968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573835.675761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573836.251694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573836.699783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573837.275708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573837.723814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573838.299682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573838.747776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573839.323702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573839.771834:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573840.347721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573840.347758:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.347765:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573840.347770:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.347781:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10593 00000100:00080000:0.0:1788573840.347792:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.347800:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573840.347839:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569764096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573840.347869:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.347873:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573840.347875:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.347879:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10593 00000100:00080000:0.0:1788573840.347882:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.347886:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573840.347896:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573840.347908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569764224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573840.347917:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569764096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573840.347921:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.347925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573840.347927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.347931:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10593 00000100:00080000:0.0:1788573840.347934:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.347937:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573840.347941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573840.347951:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569764352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573840.347959:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.347962:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573840.347964:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.347968:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10593 00000100:00080000:0.0:1788573840.347970:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.347973:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573840.347978:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573840.347989:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569764480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573840.347996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573840.348003:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573840.348007:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573840.348009:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573840.348014:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10593 00000100:00080000:0.0:1788573840.348017:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573840.348020:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573840.348023:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573840.348029:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569764608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573840.348107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff920a48353b80 x1875444569764096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573840.348125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569764096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573840.348134:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573840.348137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573840.348140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569764096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573840.348163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573840.348182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573840.348192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff920a48353800 x1875444569764224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573840.348320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569764224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573840.348327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573840.348331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573840.348333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573840.348338:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573840.348390:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573840.348402:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff920a48351c00 x1875444569764352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573840.348413:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569764352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573840.348422:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573840.348425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573840.348429:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573840.348435:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573840.348457:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573840.348469:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff920a48351500 x1875444569764480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573840.348490:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569764480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573840.348499:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573840.348502:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573840.348505:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573840.348511:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573840.348531:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573840.348552:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573840/real 1788573840] req@ffff920a48350a80 x1875444569764608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573840.348565:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569764608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573865 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573840.348573:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573840.348576:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573840.348580:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573840.795799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573841.371736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573841.819798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573842.395708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573842.843850:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573843.419731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573843.867784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573844.443706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573844.891799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573845.467704:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573845.467710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573845.467715:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573845.467721:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573845.467732:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10598 00000100:00080000:0.0:1788573845.467743:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573845.467750:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573845.467792:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569764736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573845.467817:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573845.467820:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573845.467821:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573845.467824:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10598 00000100:00080000:0.0:1788573845.467828:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573845.467831:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573845.467841:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573845.467851:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569764736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573845.467852:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569764864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573845.467866:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573845.467868:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573845.467869:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573845.467872:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10598 00000100:00080000:0.0:1788573845.467874:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573845.467877:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573845.467880:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573845.467888:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569764992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573845.467895:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573845.467898:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573845.467899:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573845.467901:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10598 00000100:00080000:0.0:1788573845.467904:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573845.467907:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573845.467910:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573845.467918:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569765120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573845.467930:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573845.467932:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573845.467934:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573845.467939:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10598 00000800:00000100:3.0:1788573845.467941:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573845.467942:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573845.467947:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573845.467950:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573845.467957:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569765248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573845.467986:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff920a48350700 x1875444569764736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.468002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569764736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.468016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.468019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573845.468022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569764736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573845.468044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573845.468061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573845.468070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff920a48350380 x1875444569764864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.468078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569764864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.468083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.468086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573845.468088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573845.468092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573845.468103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573845.468108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff920a48353480 x1875444569764992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.468115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569764992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.468121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.468123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573845.468125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569764992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573845.468129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573845.468145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573845.468156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff920a48350a80 x1875444569765120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.468169:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569765120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.468178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.468182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573845.468186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573845.468192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573845.468211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573845.468219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573845/real 1788573845] req@ffff920a48351500 x1875444569765248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573845.468227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569765248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573870 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573845.468233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573845.468235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573845.468237:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573845.915760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573846.491684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573846.939773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573847.515731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573847.963746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573848.539671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573848.987784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573849.563740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573850.011749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573850.587679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573850.587693:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573850.587699:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573850.587703:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573850.587715:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10603 00000100:00080000:0.0:1788573850.587727:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573850.587735:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573850.587783:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569765376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573850.587811:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573850.587814:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573850.587816:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573850.587820:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10603 00000100:00080000:0.0:1788573850.587823:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573850.587826:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573850.587837:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573850.587846:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569765376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573850.587852:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569765504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573850.587861:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573850.587863:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573850.587864:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573850.587867:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10603 00000100:00080000:0.0:1788573850.587869:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573850.587872:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573850.587875:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573850.587882:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569765632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573850.587888:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573850.587891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573850.587893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573850.587895:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10603 00000100:00080000:0.0:1788573850.587906:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573850.587909:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573850.587913:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573850.587920:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569765760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573850.587925:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573850.587933:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573850.587936:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573850.587938:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573850.587942:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10603 00000100:00080000:0.0:1788573850.587945:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573850.587948:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573850.587951:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573850.587958:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569765888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573850.587974:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff920a48351c00 x1875444569765376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573850.587991:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569765376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573850.588000:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573850.588013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573850.588018:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569765376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573850.588049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573850.588084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573850.588099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff920a48353800 x1875444569765504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573850.588114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569765504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573850.588124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573850.588128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573850.588132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573850.588139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573850.588156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573850.588164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff920a48353b80 x1875444569765632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573850.588172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569765632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573850.588182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573850.588185:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573850.588187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573850.588191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573850.588202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573850.588207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff920a48351500 x1875444569765760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573850.588214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569765760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573850.588220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573850.588222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573850.588224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573850.588235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573850.588247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573850.588253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573850/real 1788573850] req@ffff920a48350a80 x1875444569765888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573850.588259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569765888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573875 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573850.588264:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573850.588266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573850.588269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569765888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573851.035747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573851.611732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573852.059763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573852.635664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573853.083740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573853.659678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573854.107781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573854.683749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573855.131790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573855.707677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573855.707681:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573855.707692:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573855.707707:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573855.707721:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10609 00000100:00080000:0.0:1788573855.707735:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573855.707748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573855.707792:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569766016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573855.707817:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573855.707820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573855.707821:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573855.707824:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10609 00000100:00080000:0.0:1788573855.707827:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573855.707830:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573855.707840:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573855.707850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569766144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573855.707853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569766016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573855.707870:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573855.707873:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573855.707874:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573855.707878:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10609 00000100:00080000:0.0:1788573855.707880:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573855.707882:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573855.707885:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573855.707894:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569766272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573855.707903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573855.707907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573855.707909:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573855.707913:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10609 00000100:00080000:0.0:1788573855.707927:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573855.707932:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573855.707938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788573855.707939:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573855.707958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569766400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573855.707967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573855.707970:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573855.707972:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573855.707980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573855/real 1788573855] req@ffff920a48353480 x1875444569766016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573855.707984:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10609 00000100:00080000:0.0:1788573855.707991:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573855.707997:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573855.708002:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573855.708003:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569766016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573855.708013:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569766528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573855.708014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573855.708022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573855.708028:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569766016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573855.708052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573855.708068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573855.708076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573855/real 1788573855] req@ffff920a48350380 x1875444569766144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573855.708084:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569766144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573855.708091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573855.708094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573855.708096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573855.708100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573855.708111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573855.708117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573855/real 1788573855] req@ffff920a48350700 x1875444569766272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573855.708124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569766272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573855.708129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573855.708131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573855.708133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573855.708136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573855.708147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573855.708152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573855/real 1788573855] req@ffff920a48350a80 x1875444569766400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573855.708159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569766400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573855.708165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573855.708167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573855.708169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573855.708172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573855.708183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573855.708189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573855/real 1788573855] req@ffff920a48351500 x1875444569766528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573855.708197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569766528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573880 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573855.708202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573855.708203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573855.708205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573856.155749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573856.731702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573857.179762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573857.755645:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573858.203757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573858.779698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573859.227767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573859.803687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573860.251795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573860.827684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573860.827692:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573860.827698:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573860.827702:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573860.827710:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10614 00000100:00080000:0.0:1788573860.827719:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573860.827724:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573860.827754:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569766656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573860.827781:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573860.827785:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573860.827787:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573860.827793:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10614 00000100:00080000:0.0:1788573860.827797:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573860.827802:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573860.827812:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573860.827828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569766784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573860.827833:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569766656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573860.827850:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573860.827855:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573860.827858:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573860.827864:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10614 00000100:00080000:0.0:1788573860.827870:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573860.827883:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573860.827888:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573860.827896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573860.827901:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569766912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573860.827910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573860.827913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573860.827915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573860.827918:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10614 00000100:00080000:0.0:1788573860.827922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573860.827925:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573860.827929:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573860.827931:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573860/real 1788573860] req@ffff920a48353b80 x1875444569766656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573860.827940:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569767040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573860.827947:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573860.827950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569766656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573860.827951:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573860.827954:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573860.827962:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10614 00000100:00080000:0.0:1788573860.827966:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573860.827967:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573860.827970:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788573860.827975:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:0.0:1788573860.827976:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573860.827981:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569766656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573860.827987:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569767168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573860.828008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573860.828024:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573860.828033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573860/real 1788573860] req@ffff920a48353800 x1875444569766784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573860.828041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569766784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573860.828047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573860.828049:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573860.828052:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573860.828055:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573860.828065:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573860.828071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573860/real 1788573860] req@ffff920a48351c00 x1875444569766912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573860.828078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569766912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573860.828087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573860.828090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573860.828093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569766912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573860.828099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573860.828120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573860.828132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573860/real 1788573860] req@ffff920a48351500 x1875444569767040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573860.828145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569767040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573860.828154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573860.828157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573860.828160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573860.828167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573860.828188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573860.828199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573860/real 1788573860] req@ffff920a48350a80 x1875444569767168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573860.828210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569767168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573885 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573860.828215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573860.828217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573860.828219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573861.275766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573861.851726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573862.299815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573862.875704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573863.323789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573863.899744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573864.347761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573864.923704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573865.371827:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573865.947741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573865.947787:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573865.947794:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573865.947799:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573865.947812:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10619 00000100:00080000:0.0:1788573865.947826:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573865.947836:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573865.947902:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569767296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573865.947935:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573865.947939:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573865.947942:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573865.947947:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10619 00000100:00080000:0.0:1788573865.947952:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573865.947959:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573865.947980:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573865.947981:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569767296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573865.948002:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569767424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573865.948017:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573865.948021:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573865.948023:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573865.948027:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10619 00000100:00080000:0.0:1788573865.948045:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573865.948051:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573865.948057:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573865.948065:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573865.948073:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569767552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573865.948084:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573865.948087:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573865.948089:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573865.948093:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10619 00000100:00080000:0.0:1788573865.948096:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573865.948100:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573865.948105:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573865.948112:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573865/real 1788573865] req@ffff920a48350700 x1875444569767296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573865.948116:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569767680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573865.948148:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573865.948152:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573865.948153:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573865.948156:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10619 00000100:00080000:0.0:1788573865.948159:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573865.948161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569767296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573865.948163:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573865.948169:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573865.948180:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569767808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573865.948183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573865.948194:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573865.948215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569767296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573865.948252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573865.948278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573865.948293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573865/real 1788573865] req@ffff920a48350380 x1875444569767424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573865.948306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569767424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573865.948315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573865.948319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573865.948322:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573865.948329:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573865.948347:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573865.948358:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573865/real 1788573865] req@ffff920a48353480 x1875444569767552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573865.948369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569767552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573865.948378:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573865.948382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573865.948385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573865.948392:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573865.948411:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573865.948423:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573865/real 1788573865] req@ffff920a48350a80 x1875444569767680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573865.948434:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569767680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573865.948442:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573865.948446:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573865.948449:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573865.948455:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573865.948474:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573865.948483:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573865/real 1788573865] req@ffff920a48351500 x1875444569767808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573865.948494:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569767808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573890 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573865.948502:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573865.948505:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573865.948508:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569767808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573866.395737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573866.971747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573867.419776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573867.995652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573868.443753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573869.019679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573869.467746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573870.043673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573870.491796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573871.067702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573871.067724:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.067730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573871.067734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.067743:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10624 00000100:00080000:3.0:1788573871.067756:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.067763:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573871.067797:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569767936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.067817:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.067820:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.067821:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.067824:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10624 00000100:00080000:3.0:1788573871.067827:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.067829:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573871.067837:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.067846:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569768064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573871.067855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569767936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573871.067856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.067863:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573871.067866:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.067869:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10624 00000100:00080000:3.0:1788573871.067871:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.067874:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573871.067877:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573871.067886:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569768192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.067897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.067900:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573871.067902:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.067904:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10624 00000100:00080000:3.0:1788573871.067906:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.067909:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573871.067912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573871.067919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569768320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573871.067924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573871.067929:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788573871.067931:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573871.067932:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573871.067938:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10624 00000100:00080000:3.0:1788573871.067940:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573871.067943:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573871.067946:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573871.067955:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569768448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573871.067979:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff920a529b2d80 x1875444569767936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.067999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569767936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.068016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.068020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573871.068024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569767936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573871.068047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573871.068072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573871.068086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff920a529b2a00 x1875444569768064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.068131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569768064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.068140:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.068144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573871.068148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573871.068155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573871.068177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573871.068190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff920a529b3480 x1875444569768192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.068202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569768192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.068209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.068212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573871.068216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573871.068222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573871.068251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573871.068264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff920a529b1c00 x1875444569768320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.068277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569768320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.068285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.068288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573871.068292:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573871.068298:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573871.068320:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573871.068332:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573871/real 1788573871] req@ffff920a529b3100 x1875444569768448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573871.068352:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569768448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573896 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573871.068361:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573871.068364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573871.068367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573871.515769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573872.091675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573872.539785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573873.115678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573873.563730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573874.139691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573874.587777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573875.163672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573875.611762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573876.187721:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573876.187731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573876.187732:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573876.187739:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573876.187751:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10629 00000100:00080000:3.0:1788573876.187763:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573876.187771:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573876.187807:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569768576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573876.187835:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573876.187839:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573876.187842:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573876.187847:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10629 00000100:00080000:3.0:1788573876.187852:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573876.187857:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573876.187870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569768576:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788573876.187870:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573876.187891:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569768704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573876.187905:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573876.187910:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573876.187912:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573876.187918:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10629 00000100:00080000:3.0:1788573876.187923:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573876.187942:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573876.187952:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573876.187965:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569768832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573876.187972:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768704:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573876.187984:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573876.188002:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573876.188005:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573876.188010:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10629 00000100:00080000:3.0:1788573876.188015:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573876.188020:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573876.188027:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573876.188043:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573876.188044:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569768960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573876.188070:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573876.188074:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573876.188077:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573876.188082:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10629 00000100:00100000:1.0:1788573876.188085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768960:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788573876.188088:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573876.188152:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573876.188159:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573876.188174:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569769088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573876.188236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573876.635800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573877.211860:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573877.659728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573878.235690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573878.683776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573879.259711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788573879.260065:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788573879.260126:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788573879.260149:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573879.260201:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573879.260214:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573879.260225:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788573879.260236:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573879.260248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573879] req@ffff920a529b1f80 x1875444569768960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573879.260285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569768960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573879.260298:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573879.260304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573879.260309:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573879.260318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573879] req@ffff920a529b0000 x1875444569768832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573879.260332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569768832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573879.260341:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573879.260345:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573879.260350:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573879.260357:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573879] req@ffff920a529b0700 x1875444569768704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573879.260369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569768704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573879.260377:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573879.260382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573879.260386:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569768704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573879.260393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573879] req@ffff920a529b2300 x1875444569768576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573879.260407:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569768576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573879.260414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573879.260419:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573879.260422:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569768576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573879.260453:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573876/real 1788573879] req@ffff920a529b0e00 x1875444569769088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573879.260464:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569769088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573901 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573879.260473:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573879.260476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573879.260480:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573879.707781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573880.283713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573880.731757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788573881.307744:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573881.307756:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573881.307761:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573881.307762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573881.307774:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10634 00000100:00080000:3.0:1788573881.307786:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573881.307797:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573881.307836:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569769216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573881.307866:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573881.307871:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573881.307874:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573881.307879:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10634 00000100:00080000:3.0:1788573881.307883:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573881.307916:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569769216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573881.307923:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573881.307936:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573881.307956:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569769344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573881.307966:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573881.307970:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573881.307971:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573881.307975:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10634 00000100:00080000:3.0:1788573881.307979:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573881.307982:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788573881.307989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573881.307990:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573881.308006:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569769472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573881.308016:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573881.308020:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573881.308022:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573881.308027:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10634 00000100:00080000:3.0:1788573881.308031:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573881.308033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff920a529b3b80 x1875444569769216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573881.308037:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573881.308047:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573881.308061:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569769600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573881.308064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569769216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573881.308078:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573881.308081:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573881.308083:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573881.308088:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10634 00000100:00080000:3.0:1788573881.308094:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573881.308101:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573881.308109:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573881.308120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573881.308127:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569769728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573881.308129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573881.308140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569769216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573881.308174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573881.308206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573881.308222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff920a529b1500 x1875444569769344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.308237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569769344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.308247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.308251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573881.308255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573881.308262:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573881.308285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573881.308298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff920a529b0380 x1875444569769472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.308312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569769472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.308321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.308325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573881.308328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573881.308345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573881.308370:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573881.308383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff920a529b1180 x1875444569769600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.308397:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569769600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.308405:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.308409:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573881.308413:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573881.308419:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573881.308443:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573881.308457:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573881/real 1788573881] req@ffff920a529b0e00 x1875444569769728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573881.308480:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569769728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573906 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573881.308489:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573881.308492:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573881.308496:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573881.755748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573882.331702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573882.779808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788573883.355686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573883.803730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573884.379721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573884.827768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573885.403669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573885.851757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573886.427680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573886.427707:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.427714:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573886.427726:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.427737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10639 00000100:00080000:3.0:1788573886.427748:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.427756:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573886.427815:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569769856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.427845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.427849:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573886.427852:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.427857:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10639 00000100:00080000:3.0:1788573886.427861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.427867:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573886.427879:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573886.427894:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569769856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573886.427896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569769984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.427910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.427912:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573886.427913:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.427917:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10639 00000100:00080000:3.0:1788573886.427920:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.427923:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573886.427927:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573886.427935:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569770112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573886.427947:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.427949:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573886.427951:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.427953:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10639 00000100:00080000:3.0:1788573886.427956:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.427958:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573886.427962:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573886.427969:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569770240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573886.427974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573886.427982:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573886.427985:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573886.427987:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573886.427991:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10639 00000100:00080000:3.0:1788573886.427994:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573886.427996:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573886.428000:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573886.428006:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569770368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573886.428015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff920a529b2300 x1875444569769856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.428033:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569769856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.428042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.428046:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573886.428049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569769856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573886.428072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573886.428088:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573886.428097:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff920a529b0700 x1875444569769984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.428106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569769984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.428111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.428113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573886.428115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569769984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573886.428119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573886.428129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573886.428135:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff920a529b0000 x1875444569770112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.428147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569770112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.428153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.428154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573886.428156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573886.428160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573886.428170:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573886.428175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff920a529b1f80 x1875444569770240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.428182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569770240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.428187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.428189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573886.428192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573886.428195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573886.428205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573886.428210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573886/real 1788573886] req@ffff920a529b3100 x1875444569770368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573886.428216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569770368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573911 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573886.428226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573886.428234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573886.428236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573886.875824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573887.451733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573887.899802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573888.475702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573888.923777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573889.499657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573889.947704:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573890.523667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573890.971774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573891.547736:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573891.547737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573891.547746:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573891.547751:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573891.547760:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10644 00000100:00080000:1.0:1788573891.547769:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.547775:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573891.547815:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569770496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573891.547837:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573891.547840:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573891.547841:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573891.547844:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10644 00000100:00080000:1.0:1788573891.547847:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.547850:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573891.547858:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573891.547868:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569770624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573891.547873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569770496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573891.547889:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573891.547896:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573891.547898:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573891.547903:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10644 00000100:00080000:1.0:1788573891.547905:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.547908:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573891.547912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573891.547921:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569770752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573891.547928:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573891.547931:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573891.547932:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573891.547935:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10644 00000800:00000100:3.0:1788573891.547938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573891.547939:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.547946:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573891.547949:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573891.547959:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569770880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573891.547966:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573891.547969:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573891.547970:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573891.547974:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10644 00000100:00100000:3.0:1788573891.547974:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff920a52891f80 x1875444569770496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573891.547979:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573891.547992:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573891.547996:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573891.548001:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569770496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573891.548006:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569771008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573891.548016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573891.548021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573891.548025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569770496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573891.548045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573891.548061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573891.548070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff920a52892a00 x1875444569770624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573891.548078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569770624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573891.548083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573891.548085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573891.548087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573891.548091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573891.548102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573891.548107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff920a52893800 x1875444569770752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573891.548114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569770752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573891.548124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573891.548126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573891.548128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573891.548131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573891.548142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573891.548147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff920a52893480 x1875444569770880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573891.548160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569770880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573891.548165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573891.548167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573891.548169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569770880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573891.548173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573891.548184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573891.548189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573891/real 1788573891] req@ffff920a52890380 x1875444569771008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573891.548196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569771008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573916 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573891.548201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573891.548209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573891.548212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573891.995806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573892.571684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573893.019769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573893.595685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573894.043796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573894.619682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573895.067742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573895.643725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573896.091754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573896.667692:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788573896.667694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573896.667704:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573896.667709:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573896.667719:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10650 00000100:00080000:1.0:1788573896.667730:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573896.667737:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573896.667780:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569771136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573896.667805:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573896.667808:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573896.667810:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573896.667813:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10650 00000100:00080000:1.0:1788573896.667816:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573896.667818:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573896.667827:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573896.667836:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569771264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573896.667856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573896.667859:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573896.667860:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573896.667864:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10650 00000100:00080000:1.0:1788573896.667868:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573896.667872:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573896.667878:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573896.667880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569771136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573896.667891:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569771392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573896.667901:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573896.667903:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573896.667905:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573896.667908:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10650 00000100:00080000:1.0:1788573896.667910:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573896.667912:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573896.667916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573896.667923:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569771520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573896.667933:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573896.667936:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573896.667937:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573896.667939:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10650 00000100:00080000:1.0:1788573896.667942:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573896.667947:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573896.667951:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573896.667961:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569771648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788573896.667962:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573896.668014:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573896/real 1788573896] req@ffff920a52890e00 x1875444569771136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573896.668036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569771136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573896.668049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573896.668054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573896.668058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569771136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573896.668067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573896.668092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573896.668106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573896/real 1788573896] req@ffff920a52891180 x1875444569771264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573896.668120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569771264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573896.668130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573896.668133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573896.668137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573896.668162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573896.668182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573896.668194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573896/real 1788573896] req@ffff920a52892300 x1875444569771392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573896.668207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569771392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573896.668216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573896.668220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573896.668224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573896.668231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573896.668250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573896.668260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573896/real 1788573896] req@ffff920a52892d80 x1875444569771520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573896.668273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569771520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573896.668282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573896.668286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573896.668289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573896.668296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573896.668315:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573896.668325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573896/real 1788573896] req@ffff920a52891880 x1875444569771648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573896.668338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569771648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573921 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573896.668348:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573896.668352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573896.668356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573897.115768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573897.691816:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573898.139760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573898.715727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573899.163792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573899.739716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573900.187780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573900.763702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573901.211785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573901.787691:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573901.787700:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573901.787704:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573901.787714:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10655 00000100:00100000:2.0:1788573901.787721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573901.787724:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573901.787733:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573901.787768:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569771776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573901.787791:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573901.787794:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573901.787796:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573901.787799:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10655 00000100:00080000:1.0:1788573901.787801:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573901.787804:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573901.787813:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573901.787825:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569771904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573901.787827:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569771776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573901.787842:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573901.787845:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573901.787847:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573901.787851:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10655 00000100:00080000:1.0:1788573901.787853:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573901.787855:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573901.787858:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573901.787869:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569772032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573901.787875:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573901.787878:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573901.787879:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573901.787881:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10655 00000100:00080000:1.0:1788573901.787883:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573901.787886:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788573901.787891:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788573901.787893:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573901.787903:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569772160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573901.787910:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573901.787913:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573901.787914:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573901.787918:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10655 00000100:00080000:1.0:1788573901.787921:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573901.787926:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573901.787928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573901/real 1788573901] req@ffff920a52890700 x1875444569771776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573901.787930:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573901.787941:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569772288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573901.787949:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569771776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573901.787959:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573901.787962:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573901.787965:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569771776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573901.787984:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573901.788000:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573901.788008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573901/real 1788573901] req@ffff920a52893100 x1875444569771904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573901.788016:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569771904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573901.788022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573901.788024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573901.788026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569771904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573901.788031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573901.788042:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573901.788048:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573901/real 1788573901] req@ffff920a52891880 x1875444569772032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573901.788055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569772032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573901.788061:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573901.788062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573901.788064:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573901.788067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573901.788086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573901.788096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573901/real 1788573901] req@ffff920a52892d80 x1875444569772160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573901.788104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569772160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573901.788110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573901.788112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573901.788114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573901.788117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573901.788129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573901.788134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573901/real 1788573901] req@ffff920a52892300 x1875444569772288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573901.788147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569772288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573926 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573901.788155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573901.788158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573901.788162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573902.235754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573902.811706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573903.259766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573903.835734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573904.283761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573904.859681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573905.307739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573905.883691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573906.331747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573906.907743:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573906.907755:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573906.907759:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573906.907772:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10660 00000100:00080000:1.0:1788573906.907786:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573906.907792:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573906.907794:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573906.907850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569772416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573906.907875:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573906.907879:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573906.907881:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573906.907885:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10660 00000100:00080000:1.0:1788573906.907891:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573906.907896:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573906.907909:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573906.907912:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569772416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573906.907928:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569772544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573906.907950:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573906.907954:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573906.907957:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573906.907962:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10660 00000100:00080000:1.0:1788573906.907966:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573906.907971:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573906.907976:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573906.907983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573906.907989:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569772672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573906.907997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573906.908000:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573906.908002:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573906.908005:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10660 00000100:00080000:1.0:1788573906.908008:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573906.908011:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573906.908015:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573906.908023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573906/real 1788573906] req@ffff920a52891180 x1875444569772416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573906.908028:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569772800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573906.908047:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573906.908050:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573906.908052:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788573906.908056:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569772416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573906.908057:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10660 00000100:00080000:1.0:1788573906.908062:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573906.908068:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573906.908076:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573906.908078:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573906.908084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573906.908086:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569772928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788573906.908091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569772416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573906.908117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573906.908135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573906.908144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573906/real 1788573906] req@ffff920a52890e00 x1875444569772544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573906.908152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569772544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573906.908162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573906.908165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573906.908167:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573906.908170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573906.908181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573906.908187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573906/real 1788573906] req@ffff920a52890380 x1875444569772672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573906.908194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569772672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573906.908199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573906.908201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573906.908203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573906.908207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573906.908217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573906.908223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573906/real 1788573906] req@ffff920a52893480 x1875444569772800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573906.908230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569772800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573906.908235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573906.908236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573906.908239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573906.908243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573906.908253:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573906.908258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573906/real 1788573906] req@ffff920a52893800 x1875444569772928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573906.908265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569772928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573931 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573906.908270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573906.908272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573906.908274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569772928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573907.355733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573907.931693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573908.379739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573908.955660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573909.403768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573909.979684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573910.427789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573911.003691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573911.451774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573912.027682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573912.027695:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.027703:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573912.027708:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.027722:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10665 00000100:00080000:3.0:1788573912.027734:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.027744:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573912.027789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569773056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573912.027822:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.027826:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.027829:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.027834:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10665 00000100:00080000:3.0:1788573912.027838:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.027845:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573912.027855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569773056:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788573912.027858:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.027880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569773184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573912.027892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.027897:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573912.027899:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.027905:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10665 00000100:00080000:3.0:1788573912.027911:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.027925:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573912.027931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573912.027948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569773312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573912.027949:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573912.027992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff920a529b1c00 x1875444569773056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.028008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569773056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.028018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.028021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573912.028025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569773056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573912.028029:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.028035:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573912.028038:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.028044:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10665 00000100:00100000:1.0:1788573912.028049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773184:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573912.028050:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.028059:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573912.028067:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573912.028077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573912.028086:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569773440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573912.028091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff920a529b3480 x1875444569773184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573912.028114:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573912.028119:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573912.028121:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573912.028126:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10665 00000100:00100000:1.0:1788573912.028131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569773184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573912.028133:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573912.028141:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573912.028148:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573912.028161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.028166:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573912.028171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573912.028178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573912.028198:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573912.028207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff920a529b2a00 x1875444569773312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.028215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569773312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.028221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.028224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573912.028226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573912.028240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573912.028256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573912.028264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff920a529b2d80 x1875444569773440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.028278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569773440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.028288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788573912.028293:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569773568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573912.028296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573912.028307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573912.028329:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573912.028372:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573912.028389:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573912/real 1788573912] req@ffff920a529b1c00 x1875444569773568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573912.028401:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569773568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573937 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573912.028408:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573912.028412:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573912.028415:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573912.475714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573913.051714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573913.499767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573914.075656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573914.523714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573915.099696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573915.547747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573916.123651:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573916.571700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573917.147705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573917.147792:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.147803:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573917.147808:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.147821:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10670 00000100:00080000:3.0:1788573917.147835:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.147849:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573917.147901:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569773696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.147939:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.147945:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573917.147947:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.147952:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10670 00000100:00080000:3.0:1788573917.147956:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.147960:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573917.147973:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573917.147988:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569773824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573917.147994:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569773696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573917.148002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.148006:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573917.148007:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.148012:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10670 00000100:00080000:3.0:1788573917.148014:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.148017:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573917.148021:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573917.148029:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569773952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.148036:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.148039:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573917.148040:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.148042:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10670 00000100:00080000:3.0:1788573917.148045:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.148047:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573917.148051:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573917.148057:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569774080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573917.148063:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573917.148065:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573917.148067:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573917.148069:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10670 00000100:00080000:3.0:1788573917.148073:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573917.148075:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573917.148079:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788573917.148085:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573917.148089:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569774208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573917.148140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff920a529b3100 x1875444569773696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.148158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569773696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.148168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.148171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573917.148175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569773696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573917.148200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573917.148220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573917.148229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff920a529b1f80 x1875444569773824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.148238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569773824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.148243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.148246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573917.148248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573917.148252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573917.148264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573917.148270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff920a529b0000 x1875444569773952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.148277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569773952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.148282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.148284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573917.148286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569773952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573917.148289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573917.148300:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573917.148306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff920a529b0700 x1875444569774080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.148317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569774080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.148322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.148324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573917.148326:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573917.148329:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573917.148344:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573917.148354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573917/real 1788573917] req@ffff920a529b2300 x1875444569774208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573917.148366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569774208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573942 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573917.148374:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573917.148378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573917.148382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573917.595714:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573918.171696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573918.619769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573919.195722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573919.643744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573920.219693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573920.667749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573921.243719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573921.691799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573922.267695:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573922.267707:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573922.267712:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573922.267713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573922.267723:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10675 00000100:00080000:0.0:1788573922.267736:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573922.267745:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573922.267791:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569774336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573922.267817:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573922.267820:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573922.267822:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573922.267825:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10675 00000100:00080000:0.0:1788573922.267827:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573922.267829:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573922.267839:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573922.267850:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569774464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573922.267857:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573922.267860:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573922.267861:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573922.267864:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10675 00000100:00080000:0.0:1788573922.267866:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573922.267870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569774336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573922.267871:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573922.267878:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573922.267884:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569774592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573922.267890:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573922.267893:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573922.267894:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573922.267897:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10675 00000100:00080000:0.0:1788573922.267899:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573922.267901:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573922.267905:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573922.267912:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569774720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573922.267918:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573922.267921:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573922.267922:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573922.267925:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10675 00000100:00080000:0.0:1788573922.267941:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573922.267944:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573922.267947:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573922.267955:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569774848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573922.267965:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573922.268012:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff920a48351c00 x1875444569774336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.268041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569774336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.268052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.268057:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573922.268061:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569774336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573922.268095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.268120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573922.268135:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff920a48353800 x1875444569774464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.268155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569774464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.268165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.268169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573922.268173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573922.268179:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573922.268202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573922.268215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff920a48353b80 x1875444569774592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.268227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569774592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.268242:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.268246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573922.268249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573922.268255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573922.268277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573922.268289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff920a48351500 x1875444569774720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.268301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569774720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.268310:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.268313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573922.268317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573922.268324:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573922.268346:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573922.268357:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573922/real 1788573922] req@ffff920a48350a80 x1875444569774848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573922.268370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569774848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573947 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573922.268377:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573922.268381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573922.268385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569774848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573922.715766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573923.291693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573923.739756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573924.315698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573924.763774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573925.339702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573925.787773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573926.363669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573926.811732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573927.387686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573927.387722:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573927.387730:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573927.387734:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573927.387746:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10680 00000100:00080000:1.0:1788573927.387757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573927.387766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573927.387805:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569774976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573927.387833:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573927.387837:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573927.387840:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573927.387845:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10680 00000100:00080000:1.0:1788573927.387850:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573927.387855:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573927.387862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569774976:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788573927.387866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573927.387884:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569775104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573927.387897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573927.387901:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573927.387904:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573927.387908:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10680 00000100:00080000:1.0:1788573927.387914:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573927.387918:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573927.387924:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788573927.387933:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573927.387940:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569775232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573927.387952:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573927.387956:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573927.387958:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573927.387964:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10680 00000100:00080000:1.0:1788573927.387969:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788573927.387970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573927] req@ffff920a52892a00 x1875444569774976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573927.387976:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573927.387992:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569774976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573927.387996:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788573927.388003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573927.388007:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573927.388011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569774976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573927.388015:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569775360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573927.388028:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788573927.388033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775104:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788573927.388034:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573927.388039:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573927.388045:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10680 00000100:00080000:1.0:1788573927.388051:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788573927.388057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573927.388058:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573927.388071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573927] req@ffff920a52891f80 x1875444569775104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788573927.388075:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573927.388082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569775104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573927.388090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788573927.388094:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569775488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573927.388096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573927.388103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573927.388114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788573927.388132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573927.388142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573927] req@ffff920a52891500 x1875444569775232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573927.388151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569775232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573927.388157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573927.388160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573927.388163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788573927.388168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573927.388179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573927.388184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573927] req@ffff920a52892680 x1875444569775360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573927.388191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569775360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573927.388197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573927.388199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573927.388200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788573927.388209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788573927.388219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573927.388225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573927/real 1788573927] req@ffff920a52890a80 x1875444569775488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573927.388233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569775488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573952 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573927.388238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788573927.388239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573927.388241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788573927.835803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573928.411656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573928.859761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573929.435696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573929.883747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573930.459704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573930.907789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573931.483698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788573931.931767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573932.507726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573932.507728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573932.507737:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573932.507742:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573932.507754:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10685 00000100:00080000:3.0:1788573932.507765:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573932.507773:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573932.507817:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569775616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573932.507843:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573932.507846:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573932.507847:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573932.507850:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10685 00000100:00080000:3.0:1788573932.507853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573932.507856:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573932.507866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573932.507879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569775616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788573932.507881:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569775744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573932.507895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573932.507899:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573932.507902:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573932.507907:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10685 00000100:00080000:3.0:1788573932.507911:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573932.507916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573932.507921:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573932.507935:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569775872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573932.507943:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573932.507946:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573932.507947:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573932.507953:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10685 00000800:00000100:1.0:1788573932.507955:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573932.507958:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573932.507962:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573932.507967:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573932.507976:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569776000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573932.507983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573932.507986:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573932.507987:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573932.507990:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10685 00000100:00080000:3.0:1788573932.507993:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573932.507996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff920a529b0e00 x1875444569775616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573932.507999:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573932.508005:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573932.508016:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569776128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573932.508023:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569775616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573932.508033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573932.508037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573932.508041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569775616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573932.508065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573932.508081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573932.508089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff920a529b1180 x1875444569775744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573932.508098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569775744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573932.508104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573932.508106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573932.508108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573932.508114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573932.508125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573932.508131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff920a529b0380 x1875444569775872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573932.508139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569775872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573932.508144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573932.508146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573932.508148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569775872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573932.508151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573932.508162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573932.508167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff920a529b1500 x1875444569776000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573932.508174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569776000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573932.508179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573932.508181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573932.508183:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573932.508187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573932.508196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573932.508202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573932/real 1788573932] req@ffff920a529b3b80 x1875444569776128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573932.508209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569776128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573957 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573932.508215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573932.508216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573932.508218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573932.955809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573933.531692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573933.979755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573934.555739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573935.003790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573935.579696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573936.027807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573936.603703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573937.051804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573937.627741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573937.627741:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573937.627754:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573937.627761:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573937.627776:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10691 00000100:00080000:3.0:1788573937.627793:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573937.627804:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573937.627859:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569776256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573937.627885:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573937.627888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573937.627890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573937.627893:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10691 00000100:00080000:3.0:1788573937.627896:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573937.627899:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573937.627908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573937.627921:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569776384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573937.627928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573937.627932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573937.627933:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573937.627938:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10691 00000100:00080000:3.0:1788573937.627941:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573937.627943:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573937.627946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573937.627955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569776512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573937.627961:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569776256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573937.627964:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573937.627967:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573937.627973:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573937.627981:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10691 00000100:00080000:3.0:1788573937.627984:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573937.627987:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573937.627990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573937.628002:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569776640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573937.628008:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573937.628011:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573937.628012:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573937.628014:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10691 00000100:00080000:3.0:1788573937.628016:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573937.628019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573937.628023:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573937.628030:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569776768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788573937.628060:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573937.628105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573937/real 1788573937] req@ffff920a529b3b80 x1875444569776256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573937.628137:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569776256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573937.628154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573937.628159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573937.628164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569776256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573937.628176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573937.628210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573937.628226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573937/real 1788573937] req@ffff920a529b1500 x1875444569776384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573937.628239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569776384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573937.628264:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573937.628268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573937.628270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573937.628292:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573937.628309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573937.628318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573937/real 1788573937] req@ffff920a529b0380 x1875444569776512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573937.628325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569776512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573937.628331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573937.628334:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573937.628336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573937.628340:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573937.628358:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573937.628365:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573937/real 1788573937] req@ffff920a529b1180 x1875444569776640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573937.628372:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569776640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573937.628377:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573937.628379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573937.628382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573937.628386:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573937.628397:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573937.628403:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573937/real 1788573937] req@ffff920a529b0e00 x1875444569776768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573937.628411:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569776768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573962 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573937.628416:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573937.628418:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573937.628419:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569776768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573938.075774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573938.651727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573939.099791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573939.675751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573940.123774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573940.699739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573941.147794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573941.723856:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573942.171732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573942.747734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573942.747739:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573942.747751:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573942.747756:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573942.747766:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10696 00000100:00080000:0.0:1788573942.747777:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573942.747786:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573942.747823:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569776896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573942.747851:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573942.747856:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573942.747858:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573942.747862:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10696 00000100:00080000:0.0:1788573942.747866:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573942.747870:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573942.747880:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573942.747896:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569777024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573942.747902:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569776896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573942.747918:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573942.747923:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573942.747926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573942.747932:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10696 00000100:00080000:0.0:1788573942.747937:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573942.747942:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573942.747947:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573942.747964:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569777152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573942.747974:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573942.747979:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573942.747982:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573942.747988:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10696 00000100:00080000:0.0:1788573942.747993:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788573942.747996:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777024:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573942.748004:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573942.748013:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573942.748032:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569777280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573942.748039:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777152:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788573942.748061:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788573942.748064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777280:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788573942.748069:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573942.748074:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573942.748081:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10696 00000100:00080000:0.0:1788573942.748087:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573942.748092:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573942.748098:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573942.748113:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569777408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573942.748163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573943.195820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573943.771810:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573944.219765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573944.795830:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573945.243778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573945.819905:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788573945.822822:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788573945.822884:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788573945.822904:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573945.822957:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573945.822975:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788573945.822985:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573945.822992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573942/real 1788573945] req@ffff920a48350380 x1875444569777024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788573945.823001:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788573945.823026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569777024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573945.823042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573945.823047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573945.823050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573945.823056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573942/real 1788573945] req@ffff920a48350700 x1875444569777152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573945.823065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569777152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573945.823070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573945.823073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573945.823075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573945.823079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573942/real 1788573945] req@ffff920a48353480 x1875444569776896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573945.823087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569776896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573945.823091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573945.823093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573945.823095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569776896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573945.823127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573942/real 1788573945] req@ffff920a48351500 x1875444569777408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573945.823135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569777408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573945.823142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573945.823144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573945.823146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777408:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573945.823150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573942/real 1788573945] req@ffff920a48350a80 x1875444569777280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573945.823159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569777280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573967 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573945.823168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573945.823171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573945.823174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777280:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573946.267709:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573946.843796:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573947.291791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573947.867788:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573947.867793:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573947.867806:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573947.867811:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573947.867823:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10701 00000100:00080000:0.0:1788573947.867836:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573947.867845:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573947.867887:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569777536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573947.867925:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573947.867930:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573947.867933:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573947.867938:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10701 00000100:00080000:0.0:1788573947.867943:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573947.867948:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573947.867954:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569777536:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788573947.867961:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573947.867983:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569777664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573947.867992:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573947.867995:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573947.867996:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573947.868001:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10701 00000100:00080000:0.0:1788573947.868022:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573947.868028:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573947.868034:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788573947.868043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573947.868048:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569777792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573947.868060:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573947.868064:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573947.868067:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573947.868071:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10701 00000100:00080000:0.0:1788573947.868077:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573947.868082:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573947.868090:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573947.868090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573947/real 1788573947] req@ffff920a48353b80 x1875444569777536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573947.868108:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569777920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573947.868120:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573947.868124:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573947.868126:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788573947.868129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569777536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573947.868133:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10701 00000100:00080000:0.0:1788573947.868140:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573947.868148:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573947.868154:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788573947.868155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573947.868163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573947.868167:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569778048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788573947.868171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569777536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573947.868210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573947.868245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573947.868260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573947/real 1788573947] req@ffff920a48353800 x1875444569777664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573947.868275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569777664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573947.868285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573947.868290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573947.868295:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573947.868304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788573947.868336:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573947.868349:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573947/real 1788573947] req@ffff920a48351c00 x1875444569777792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573947.868362:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569777792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573947.868371:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573947.868375:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573947.868380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788573947.868388:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573947.868414:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573947.868430:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573947/real 1788573947] req@ffff920a48350a80 x1875444569777920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573947.868449:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569777920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573947.868457:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573947.868460:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573947.868463:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569777920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788573947.868469:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788573947.868491:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573947.868504:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573947/real 1788573947] req@ffff920a48351500 x1875444569778048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573947.868515:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569778048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573972 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573947.868523:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788573947.868527:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573947.868531:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573948.315819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573948.891796:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573949.339749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573949.915724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573950.363719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573950.939688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573951.387737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573951.963731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573952.411757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573952.987691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573952.987715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573952.987723:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573952.987727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573952.987739:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10706 00000100:00080000:1.0:1788573952.987748:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573952.987755:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573952.987785:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569778176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573952.987823:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573952.987828:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573952.987832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573952.987839:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10706 00000100:00080000:1.0:1788573952.987844:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573952.987849:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573952.987860:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573952.987865:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569778176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573952.987883:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569778304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573952.987899:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573952.987904:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573952.987906:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573952.987911:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10706 00000100:00080000:1.0:1788573952.987917:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573952.987924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573952.987931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573952.987945:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569778432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573952.987956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573952.987959:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573952.987966:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573952.987968:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573952.987976:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10706 00000100:00080000:1.0:1788573952.987983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573952.987991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573952.987998:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573952.988004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573952/real 1788573952] req@ffff920a52890000 x1875444569778176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573952.988012:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569778560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573952.988084:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573952.988089:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573952.988092:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573952.988096:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10706 00000100:00100000:2.0:1788573952.988101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569778176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788573952.988104:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573952.988116:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573952.988123:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573952.988125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573952.988131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788573952.988136:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569778688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573952.988146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569778176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573952.988206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573952.988232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573952.988243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573952/real 1788573952] req@ffff920a52893b80 x1875444569778304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573952.988252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569778304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573952.988259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573952.988269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573952.988272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573952.988276:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573952.988290:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573952.988298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573952/real 1788573952] req@ffff920a52890a80 x1875444569778432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573952.988310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569778432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573952.988325:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573952.988329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573952.988333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573952.988339:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573952.988358:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573952.988365:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573952/real 1788573952] req@ffff920a52892680 x1875444569778560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573952.988373:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569778560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573952.988379:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573952.988382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573952.988384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573952.988390:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573952.988410:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573952.988422:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573952/real 1788573952] req@ffff920a52891500 x1875444569778688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573952.988435:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569778688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573977 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573952.988444:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573952.988446:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573952.988449:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573953.435744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573954.011738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573954.459760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573955.035715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573955.483746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573956.059724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573956.507730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573957.083677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573957.531825:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573958.107704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573958.107859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.107867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573958.107885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.107898:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10711 00000100:00080000:3.0:1788573958.107909:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.107918:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573958.107962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569778816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.107986:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.107989:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573958.107990:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.107993:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10711 00000100:00080000:3.0:1788573958.107995:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.107998:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573958.108008:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573958.108019:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569778944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.108030:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573958.108032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569778816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573958.108034:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573958.108039:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.108051:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10711 00000100:00080000:3.0:1788573958.108055:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.108057:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573958.108061:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573958.108070:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569779072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.108076:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.108080:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573958.108081:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.108084:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10711 00000100:00080000:3.0:1788573958.108086:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.108089:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573958.108093:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573958.108099:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569779200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573958.108105:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573958.108108:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573958.108109:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573958.108113:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10711 00000100:00080000:3.0:1788573958.108116:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573958.108119:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573958.108124:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573958.108132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788573958.108137:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569779328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573958.108186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff920a529b2300 x1875444569778816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.108207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569778816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.108218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.108223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573958.108227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569778816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573958.108251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573958.108274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573958.108286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff920a529b0700 x1875444569778944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.108298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569778944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.108312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.108316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573958.108319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569778944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573958.108326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573958.108344:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573958.108353:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff920a529b0000 x1875444569779072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.108365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569779072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.108374:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.108377:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573958.108381:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573958.108387:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573958.108405:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573958.108414:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff920a529b1f80 x1875444569779200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.108427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569779200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.108435:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.108438:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573958.108441:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573958.108447:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573958.108465:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573958.108475:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573958/real 1788573958] req@ffff920a529b3100 x1875444569779328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573958.108487:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569779328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573983 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573958.108494:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573958.108497:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573958.108501:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573958.555858:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573959.131710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573959.579766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573960.155684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573960.603808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573961.179717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573961.627806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573962.203708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573962.651740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573963.227709:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573963.227722:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573963.227727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573963.227739:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10716 00000100:00080000:1.0:1788573963.227753:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573963.227758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573963.227763:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573963.227819:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569779456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573963.227845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573963.227849:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573963.227850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573963.227853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10716 00000100:00080000:1.0:1788573963.227856:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573963.227860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573963.227871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573963.227880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569779584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573963.227890:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573963.227893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573963.227894:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788573963.227901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569779456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573963.227905:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10716 00000100:00080000:1.0:1788573963.227916:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573963.227920:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573963.227924:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573963.227954:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569779712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573963.227962:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573963.227965:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573963.227966:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573963.227970:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10716 00000100:00080000:1.0:1788573963.227975:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573963.227977:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573963.227984:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788573963.228002:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569779840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573963.228004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788573963.228019:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573963.228023:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573963.228025:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573963.228032:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10716 00000100:00080000:1.0:1788573963.228037:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573963.228042:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573963.228049:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573963.228049:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff920a52891f80 x1875444569779456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573963.228078:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569779968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573963.228080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569779456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.228106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.228111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573963.228116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569779456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573963.228139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573963.228162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573963.228171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff920a52892a00 x1875444569779584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.228180:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569779584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.228185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.228187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573963.228190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573963.228193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573963.228204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573963.228211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff920a52893800 x1875444569779712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.228218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569779712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.228223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.228225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573963.228226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573963.228230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573963.228242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573963.228247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff920a52893480 x1875444569779840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.228254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569779840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.228259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.228261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573963.228263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573963.228266:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573963.228276:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573963.228282:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573963/real 1788573963] req@ffff920a52890380 x1875444569779968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573963.228288:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569779968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573988 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573963.228292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573963.228295:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573963.228297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569779968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573963.675736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573964.251738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573964.699753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573965.275681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573965.723746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573966.299705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573966.747783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573967.323671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573967.771701:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573968.347688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573968.347708:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.347715:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573968.347720:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.347731:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10721 00000100:00080000:3.0:1788573968.347741:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.347749:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573968.347785:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569780096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573968.347814:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.347818:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573968.347821:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.347827:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10721 00000100:00080000:3.0:1788573968.347832:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573968.347839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569780096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573968.347840:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573968.347856:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573968.347875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569780224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573968.347886:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573968.347890:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573968.347893:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.347897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10721 00000100:00080000:3.0:1788573968.347902:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788573968.347907:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788573968.347910:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573968.347917:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573968.347943:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569780352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573968.347948:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff920a529b1c00 x1875444569780096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.347974:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573968.347979:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569780096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573968.347980:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573968.347985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.348006:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10721 00000100:00080000:3.0:1788573968.348010:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.348014:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573968.348024:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788573968.348027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.348035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788573968.348040:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569780480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573968.348042:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569780096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573968.348058:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573968.348071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780224:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788573968.348072:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573968.348078:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573968.348085:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10721 00000100:00080000:3.0:1788573968.348090:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573968.348096:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788573968.348101:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573968.348105:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573968.348113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff920a529b2d80 x1875444569780224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.348130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569780224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573968.348133:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569780608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573968.348155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.348162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573968.348167:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573968.348175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573968.348199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573968.348213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff920a529b2a00 x1875444569780352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.348226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569780352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.348235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.348239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573968.348243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573968.348249:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573968.348267:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573968.348275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff920a529b3480 x1875444569780480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.348283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569780480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.348288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.348291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573968.348293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573968.348304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573968.348315:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573968.348324:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573968/real 1788573968] req@ffff920a529b1c00 x1875444569780608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573968.348337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569780608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573993 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573968.348346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573968.348349:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573968.348352:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573968.795783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573969.371734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573969.819778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573970.395694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573970.843773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573971.419711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573971.867826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573972.443668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573972.891728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573973.467697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573973.467697:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573973.467709:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573973.467715:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573973.467729:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10726 00000100:00080000:3.0:1788573973.467740:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573973.467748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573973.467798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569780736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573973.467819:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573973.467821:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573973.467822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573973.467825:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10726 00000100:00080000:3.0:1788573973.467827:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573973.467830:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573973.467840:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573973.467849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569780864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573973.467857:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573973.467861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788573973.467862:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569780736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573973.467863:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573973.467871:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10726 00000100:00080000:3.0:1788573973.467874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573973.467876:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573973.467879:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573973.467890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569780992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573973.467906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573973.467910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573973.467913:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573973.467917:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10726 00000100:00080000:3.0:1788573973.467937:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573973.467941:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788573973.467947:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788573973.467947:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573973.467961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569781120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573973.467968:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573973.467977:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788573973.467979:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573973.467986:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10726 00000100:00100000:2.0:1788573973.467989:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff920a529b3100 x1875444569780736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788573973.467991:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573973.468000:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573973.468007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573973.468019:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569780736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788573973.468019:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569781248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788573973.468067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573973.468072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573973.468077:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569780736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573973.468101:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573973.468130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573973.468141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff920a529b1f80 x1875444569780864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573973.468155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569780864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573973.468161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573973.468163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573973.468166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573973.468169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573973.468182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573973.468188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff920a529b0000 x1875444569780992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573973.468196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569780992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573973.468201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573973.468203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573973.468205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569780992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573973.468208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573973.468220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573973.468225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff920a529b0700 x1875444569781120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573973.468233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569781120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573973.468243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573973.468245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573973.468247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573973.468250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573973.468264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573973.468273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573973/real 1788573973] req@ffff920a529b2300 x1875444569781248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573973.468284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569781248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788573998 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573973.468292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573973.468296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573973.468300:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573973.915821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573974.491665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573974.939814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573975.515762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573975.963766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573976.539699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573976.987719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573977.563727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573978.011779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573978.587684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573978.587748:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573978.587760:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573978.587765:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573978.587775:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10731 00000100:00080000:1.0:1788573978.587786:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573978.587794:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573978.587835:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569781376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573978.587871:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573978.587876:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573978.587879:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573978.587888:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10731 00000100:00080000:1.0:1788573978.587892:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573978.587896:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573978.587906:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573978.587911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569781376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788573978.587924:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569781504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573978.587936:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573978.587939:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573978.587940:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573978.587944:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10731 00000100:00080000:1.0:1788573978.587946:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573978.587949:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573978.587952:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573978.587963:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569781632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573978.587969:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573978.587971:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573978.587972:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573978.587975:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10731 00000100:00080000:1.0:1788573978.587980:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573978.587982:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573978.587987:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788573978.587995:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573978.588000:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569781760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573978.588014:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573978.588018:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573978.588019:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573978.588025:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10731 00000100:00080000:1.0:1788573978.588027:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573978.588030:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573978.588038:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788573978.588050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff920a52890e00 x1875444569781376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573978.588052:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569781888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573978.588113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569781376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573978.588125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573978.588129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573978.588132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569781376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573978.588162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573978.588182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573978.588192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff920a52891180 x1875444569781504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573978.588200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569781504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573978.588206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573978.588208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573978.588210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573978.588215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573978.588226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573978.588232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff920a52892300 x1875444569781632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573978.588239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569781632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573978.588249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573978.588251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573978.588253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573978.588257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573978.588269:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573978.588274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff920a52892d80 x1875444569781760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573978.588281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569781760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573978.588286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573978.588289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573978.588291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573978.588294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573978.588304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573978.588310:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573978/real 1788573978] req@ffff920a52891880 x1875444569781888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573978.588317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569781888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574003 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573978.588322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573978.588324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573978.588327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569781888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573979.035782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573979.611716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573980.059778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573980.635681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573981.083759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573981.659724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573982.107778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573982.683713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573983.131735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788573983.707727:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573983.707740:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788573983.707745:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573983.707759:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10737 00000100:00100000:2.0:1788573983.707760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788573983.707772:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573983.707781:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788573983.707822:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569782016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573983.707846:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573983.707850:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788573983.707852:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573983.707861:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10737 00000100:00080000:1.0:1788573983.707865:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573983.707878:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573983.707888:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573983.707899:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569782144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573983.707908:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573983.707912:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788573983.707914:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788573983.707921:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569782016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788573983.707925:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10737 00000100:00080000:1.0:1788573983.707939:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573983.707945:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573983.707950:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788573983.707965:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569782272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573983.707975:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573983.707978:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788573983.707980:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573983.707984:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10737 00000100:00080000:1.0:1788573983.707999:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573983.708004:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573983.708009:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788573983.708019:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788573983.708024:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569782400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788573983.708033:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788573983.708036:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788573983.708039:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788573983.708044:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10737 00000100:00080000:1.0:1788573983.708049:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788573983.708056:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788573983.708067:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573983.708077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573983/real 1788573983] req@ffff920a52893100 x1875444569782016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788573983.708087:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569782528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788573983.708119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569782016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573983.708139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573983.708143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573983.708146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569782016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573983.708173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573983.708194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573983.708205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573983/real 1788573983] req@ffff920a52890700 x1875444569782144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573983.708213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569782144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573983.708220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573983.708222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573983.708224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573983.708228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788573983.708243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573983.708250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573983/real 1788573983] req@ffff920a52891880 x1875444569782272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573983.708258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569782272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573983.708264:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573983.708266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573983.708268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788573983.708271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573983.708286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573983.708293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573983/real 1788573983] req@ffff920a52892d80 x1875444569782400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573983.708301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569782400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573983.708306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573983.708308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573983.708310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788573983.708315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788573983.708327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788573983.708333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573983/real 1788573983] req@ffff920a52892300 x1875444569782528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788573983.708347:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569782528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574008 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788573983.708352:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788573983.708355:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788573983.708357:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573984.155823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573984.731736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573985.179739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573985.755701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573986.203739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573986.779676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573987.227877:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573987.803743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573988.251759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573988.827724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573988.827750:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573988.827759:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573988.827764:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573988.827775:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10742 00000100:00080000:0.0:1788573988.827786:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573988.827795:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573988.827839:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569782656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573988.827866:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573988.827869:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573988.827871:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573988.827874:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10742 00000100:00080000:0.0:1788573988.827876:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573988.827879:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573988.827889:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573988.827901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569782784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573988.827910:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573988.827912:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573988.827913:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573988.827918:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10742 00000100:00080000:0.0:1788573988.827923:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788573988.827924:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569782656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788573988.827930:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573988.827939:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573988.827949:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569782912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573988.827956:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573988.827959:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573988.827961:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573988.827965:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10742 00000100:00080000:0.0:1788573988.827968:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573988.827970:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573988.827973:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573988.827982:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569783040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573988.827988:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573988.827991:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573988.827992:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573988.827996:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10742 00000100:00080000:0.0:1788573988.827998:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573988.828002:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573988.828005:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573988.828015:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569783168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573988.828026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573988.828082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573988/real 1788573988] req@ffff920a48353480 x1875444569782656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573988.828104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569782656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573988.828146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573988.828152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573988.828156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569782656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573988.828186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573988.828217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573988.828232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573988/real 1788573988] req@ffff920a48350700 x1875444569782784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573988.828245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569782784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573988.828254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573988.828259:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573988.828263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573988.828270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573988.828292:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573988.828303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573988/real 1788573988] req@ffff920a48350380 x1875444569782912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573988.828316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569782912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573988.828326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573988.828330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573988.828333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569782912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573988.828340:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573988.828363:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573988.828375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573988/real 1788573988] req@ffff920a48351500 x1875444569783040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573988.828395:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569783040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573988.828413:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573988.828417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573988.828421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573988.828428:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573988.828451:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573988.828465:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573988/real 1788573988] req@ffff920a48350a80 x1875444569783168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573988.828477:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569783168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574013 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573988.828485:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573988.828489:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573988.828492:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573989.275806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573989.851673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573990.299767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573990.875699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573991.323772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573991.899678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573992.347753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573992.923695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573993.371765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788573993.947692:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788573993.947692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788573993.947703:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788573993.947707:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573993.947718:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10747 00000100:00080000:0.0:1788573993.947730:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573993.947748:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788573993.947788:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569783296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573993.947823:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573993.947826:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788573993.947828:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573993.947832:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10747 00000100:00080000:0.0:1788573993.947835:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573993.947839:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573993.947849:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788573993.947852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569783296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788573993.947864:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569783424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573993.947873:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573993.947876:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788573993.947878:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573993.947881:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10747 00000100:00080000:0.0:1788573993.947884:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573993.947887:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573993.947891:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788573993.947899:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569783552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573993.947907:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573993.947909:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788573993.947911:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788573993.947916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788573993.947919:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10747 00000100:00080000:0.0:1788573993.947923:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573993.947926:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573993.947930:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573993.947939:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569783680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788573993.947946:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788573993.947948:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788573993.947950:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788573993.947953:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10747 00000100:00080000:0.0:1788573993.947957:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788573993.947959:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788573993.947962:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788573993.947970:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569783808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788573993.947977:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573993/real 1788573993] req@ffff920a48351c00 x1875444569783296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573993.948000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569783296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573993.948011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573993.948014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573993.948018:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569783296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573993.948042:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573993.948068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573993.948082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573993/real 1788573993] req@ffff920a48353800 x1875444569783424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573993.948097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569783424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573993.948103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573993.948105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573993.948108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573993.948112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573993.948152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573993.948163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573993/real 1788573993] req@ffff920a48353b80 x1875444569783552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573993.948176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569783552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573993.948181:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573993.948183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573993.948185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573993.948189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573993.948204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573993.948210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573993/real 1788573993] req@ffff920a48350a80 x1875444569783680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573993.948219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569783680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573993.948227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573993.948230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573993.948233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573993.948240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573993.948260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573993.948273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573993/real 1788573993] req@ffff920a48351500 x1875444569783808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573993.948285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569783808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574018 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573993.948300:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573993.948304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573993.948307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569783808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788573994.395756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573994.971703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573995.419809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573995.995690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573996.443767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788573997.019728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788573997.467739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573998.043776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788573998.491753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788573999.067715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788573999.067715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.067724:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788573999.067728:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.067736:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10752 00000100:00080000:3.0:1788573999.067746:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.067752:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788573999.067789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569783936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.067817:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.067820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788573999.067822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.067825:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10752 00000100:00080000:3.0:1788573999.067828:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.067831:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573999.067840:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573999.067852:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569784064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.067858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.067861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788573999.067862:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.067865:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10752 00000100:00080000:3.0:1788573999.067866:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.067869:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573999.067872:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788573999.067878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569784192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.067886:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.067889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788573999.067890:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.067894:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10752 00000100:00080000:3.0:1788573999.067896:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.067898:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573999.067907:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573999.067915:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569784320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788573999.067924:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788573999.067930:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788573999.067931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569783936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788573999.067933:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788573999.067945:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10752 00000100:00080000:3.0:1788573999.067951:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788573999.067955:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788573999.067961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788573999.067976:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569784448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788573999.068029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573999.068072:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff920a529b0e00 x1875444569783936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573999.068088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569783936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573999.068098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573999.068102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573999.068105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569783936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788573999.068111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573999.068127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573999.068135:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff920a529b1180 x1875444569784064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573999.068165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569784064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573999.068171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573999.068173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573999.068176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573999.068180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788573999.068195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573999.068202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff920a529b0380 x1875444569784192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573999.068210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569784192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573999.068215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573999.068217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573999.068219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788573999.068243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573999.068255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573999.068261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff920a529b1500 x1875444569784320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573999.068269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569784320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573999.068278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573999.068281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573999.068285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788573999.068291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788573999.068309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788573999.068321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788573999/real 1788573999] req@ffff920a529b3b80 x1875444569784448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788573999.068334:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569784448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574024 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788573999.068343:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788573999.068346:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788573999.068350:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788573999.515764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574000.091738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574000.539748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574001.115686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574001.563838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574002.139715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574002.587780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574003.163702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574003.611753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574004.187688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574004.187705:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.187711:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574004.187714:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.187723:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10757 00000100:00080000:2.0:1788574004.187733:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.187739:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574004.187772:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569784576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.187802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.187805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574004.187807:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.187810:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10757 00000100:00080000:2.0:1788574004.187813:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.187816:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574004.187824:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574004.187832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569784576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574004.187844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569784704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.187852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.187855:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574004.187857:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.187860:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10757 00000100:00080000:2.0:1788574004.187863:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574004.187865:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574004.187869:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574004.187878:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569784832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.187885:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.187888:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574004.187889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.187891:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10757 00000100:00080000:2.0:1788574004.187895:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574004.187899:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574004.187902:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574004.187907:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574004.187916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569784960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574004.187923:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574004.187926:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574004.187927:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574004.187930:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10757 00000100:00080000:2.0:1788574004.187933:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574004.187934:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff920a60ac4a80 x1875444569784576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574004.187939:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574004.187945:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574004.187955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569785088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574004.187959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569784576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.187979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.187983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574004.187987:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569784576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574004.188005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574004.188022:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574004.188029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff920a60ac4700 x1875444569784704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.188038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569784704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.188044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.188046:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574004.188048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574004.188052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574004.188062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574004.188068:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff920a60ac4380 x1875444569784832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.188075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569784832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.188081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.188083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574004.188084:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574004.188088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574004.188098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574004.188114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff920a60ac4000 x1875444569784960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.188122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569784960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.188127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.188129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574004.188131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569784960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574004.188134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574004.188145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574004.188151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574004/real 1788574004] req@ffff920a60ac7800 x1875444569785088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574004.188158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569785088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574029 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574004.188163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574004.188180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574004.188182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574004.635733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574005.211722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574005.659787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574006.235731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574006.683765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574007.259682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574007.707736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574008.283702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574008.731767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574009.307697:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.307708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788574009.307711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574009.307713:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.307725:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10762 00000100:00080000:2.0:1788574009.307736:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.307743:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574009.307781:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569785216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.307808:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.307812:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574009.307814:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.307819:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10762 00000100:00080000:2.0:1788574009.307823:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.307828:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574009.307838:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574009.307839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569785216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574009.307857:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569785344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.307870:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.307874:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574009.307876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.307880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10762 00000100:00080000:2.0:1788574009.307885:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.307890:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574009.307895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574009.307908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569785472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.307919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.307923:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788574009.307927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785344:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788574009.307927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.307938:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10762 00000100:00080000:2.0:1788574009.307943:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.307948:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574009.307954:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574009.307958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574009.307972:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569785600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574009.307986:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574009.307990:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574009.307993:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785600:192.168.202.157@tcp:8:kworker.0 00000100:00080000:2.0:1788574009.307994:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574009.308003:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10762 00000100:00080000:2.0:1788574009.308008:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574009.308013:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574009.308019:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574009.308031:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569785728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574009.308074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574009.755784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574010.331726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574010.779818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574011.355668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574011.803780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574012.379715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788574012.380019:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788574012.380084:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574012.380105:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574012.380155:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574012.380165:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574012.380173:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574012.380187:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574012.380248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574012] req@ffff920a60ac4380 x1875444569785728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574012.380276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569785728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574012.380288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574012.380294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574012.380300:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785728:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574012.380309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574012] req@ffff920a60ac4000 x1875444569785600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574012.380322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569785600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574012.380330:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574012.380333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574012.380337:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574012.380344:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574012] req@ffff920a60ac5c00 x1875444569785344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574012.380358:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569785344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574012.380367:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574012.380371:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574012.380375:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574012.380382:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574012] req@ffff920a60ac7800 x1875444569785472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574012.380394:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569785472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574012.380402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574012.380406:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574012.380410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574012.380417:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574009/real 1788574012] req@ffff920a60ac4e00 x1875444569785216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574012.380429:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569785216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574034 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574012.380437:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574012.380441:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574012.380444:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569785216:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788574012.827794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574013.403752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574013.851810:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574014.427744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574014.427759:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574014.427769:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574014.427774:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574014.427791:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10767 00000100:00080000:3.0:1788574014.427805:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574014.427814:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574014.427855:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569785856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574014.427880:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574014.427883:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574014.427884:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574014.427888:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10767 00000100:00080000:3.0:1788574014.427890:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574014.427893:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574014.427903:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574014.427914:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569785984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574014.427924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574014.427930:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574014.427931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569785856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574014.427933:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574014.427944:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10767 00000100:00080000:3.0:1788574014.427951:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574014.427955:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574014.427958:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574014.427968:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569786112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574014.427975:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574014.427978:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574014.427979:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574014.427982:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10767 00000100:00080000:3.0:1788574014.427986:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574014.427988:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574014.427993:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574014.428001:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569786240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574014.428008:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574014.428011:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574014.428013:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574014.428018:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10767 00000100:00080000:3.0:1788574014.428023:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574014.428033:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574014.428035:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574014.428049:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574014.428067:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569786368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574014.428084:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff920a529b3b80 x1875444569785856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.428107:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569785856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.428118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.428124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574014.428128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569785856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574014.428156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574014.428185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574014.428199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff920a529b1500 x1875444569785984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.428245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569785984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.428253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.428257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574014.428260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569785984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574014.428268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574014.428290:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574014.428302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff920a529b0380 x1875444569786112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.428314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569786112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.428322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.428325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574014.428328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574014.428335:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574014.428356:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574014.428369:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff920a529b1180 x1875444569786240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.428382:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569786240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.428391:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.428394:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574014.428397:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574014.428403:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574014.428424:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574014.428435:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574014/real 1788574014] req@ffff920a529b0e00 x1875444569786368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574014.428447:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569786368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574039 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574014.428456:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574014.428459:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574014.428462:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574014.875797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574015.451660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574015.899780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574016.475700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574016.923777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574017.499693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574017.947772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574018.523699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574018.971806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574019.547709:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574019.547714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574019.547723:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574019.547730:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574019.547742:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10772 00000100:00080000:2.0:1788574019.547755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574019.547767:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574019.547814:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569786496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574019.547844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574019.547849:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574019.547851:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574019.547855:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10772 00000100:00080000:2.0:1788574019.547859:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574019.547874:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574019.547880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569786496:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574019.547894:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574019.547914:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569786624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574019.547926:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574019.547931:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574019.547934:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574019.547938:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10772 00000100:00080000:2.0:1788574019.547943:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574019.547949:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574019.547954:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574019.547960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574019.547970:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569786752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574019.547983:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574019.547986:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574019.547987:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574019.547991:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10772 00000100:00080000:2.0:1788574019.547994:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574019.547997:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff920a60ac4700 x1875444569786496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574019.548001:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574019.548010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574019.548021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569786496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574019.548022:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569786880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574019.548052:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574019.548054:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574019.548055:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574019.548060:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574019.548064:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788574019.548066:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10772 00000100:00080000:2.0:1788574019.548070:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574019.548071:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569786496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574019.548077:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574019.548083:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574019.548089:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569787008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574019.548102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574019.548136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574019.548146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff920a60ac4a80 x1875444569786624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574019.548155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569786624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574019.548161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574019.548163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574019.548165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574019.548170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574019.548182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574019.548187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff920a60ac5500 x1875444569786752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574019.548200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569786752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574019.548206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574019.548208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574019.548210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574019.548213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574019.548242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574019.548251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff920a60ac6300 x1875444569786880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574019.548259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569786880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574019.548265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574019.548267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574019.548269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569786880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574019.548273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574019.548286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574019.548293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574019/real 1788574019] req@ffff920a60ac7100 x1875444569787008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574019.548301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569787008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574044 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574019.548311:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574019.548315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574019.548317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574019.995793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574020.571887:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574021.019777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574021.595690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574022.043749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574022.619682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574023.067769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574023.643703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574024.091734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574024.667663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574024.667733:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574024.667741:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574024.667747:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574024.667760:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10778 00000100:00080000:2.0:1788574024.667773:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574024.667783:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574024.667833:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569787136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574024.667868:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574024.667872:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574024.667874:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574024.667878:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10778 00000100:00080000:2.0:1788574024.667881:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574024.667885:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574024.667897:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574024.667911:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569787264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574024.667920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574024.667926:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574024.667928:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569787136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574024.667929:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574024.667941:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10778 00000100:00080000:2.0:1788574024.667951:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574024.667956:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574024.667960:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574024.667971:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569787392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574024.667983:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574024.667985:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574024.667987:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574024.667989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10778 00000100:00080000:2.0:1788574024.667991:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574024.667994:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574024.667999:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574024.668006:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569787520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574024.668013:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574024.668015:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574024.668021:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574024.668023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574024.668038:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10778 00000100:00080000:2.0:1788574024.668041:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574024.668044:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574024.668049:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574024.668059:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569787648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574024.668070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574024/real 1788574024] req@ffff920a60ac6a00 x1875444569787136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574024.668092:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569787136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574024.668101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574024.668104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574024.668107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569787136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574024.668127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574024.668144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574024.668152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574024/real 1788574024] req@ffff920a60ac5180 x1875444569787264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574024.668160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569787264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574024.668171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574024.668174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574024.668176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574024.668180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574024.668191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574024.668197:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574024/real 1788574024] req@ffff920a60ac7100 x1875444569787392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574024.668204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569787392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574024.668210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574024.668212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574024.668214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574024.668217:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574024.668228:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574024.668233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574024/real 1788574024] req@ffff920a60ac6300 x1875444569787520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574024.668264:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569787520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574024.668270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574024.668272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574024.668275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574024.668280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574024.668294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574024.668301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574024/real 1788574024] req@ffff920a60ac5500 x1875444569787648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574024.668309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569787648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574049 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574024.668314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574024.668316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574024.668318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574025.115751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574025.691672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574026.139753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574026.715708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574027.163796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574027.739695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574028.187792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574028.763700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574029.211753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574029.787687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574029.787723:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574029.787728:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574029.787732:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574029.787740:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10783 00000100:00080000:3.0:1788574029.787748:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574029.787754:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574029.787788:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569787776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574029.787809:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574029.787812:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574029.787814:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574029.787817:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10783 00000100:00080000:3.0:1788574029.787821:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574029.787824:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574029.787833:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574029.787844:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569787776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574029.787845:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569787904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574029.787860:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574029.787865:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574029.787867:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574029.787872:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10783 00000100:00080000:3.0:1788574029.787877:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574029.787881:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574029.787887:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574029.787901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569788032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574029.787904:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574029.787913:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574029.787917:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574029.787918:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574029.787923:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10783 00000100:00080000:3.0:1788574029.787926:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574029.787929:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574029.787934:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574029.787939:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574029/real 1788574029] req@ffff920a529b2300 x1875444569787776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574029.787943:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569788160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574029.787968:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574029.787971:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574029.787973:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574029.787975:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569787776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574029.787978:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10783 00000100:00080000:3.0:1788574029.787983:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574029.787989:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574029.787996:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574029.787997:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574029.788003:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574029.788006:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569788288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574029.788010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569787776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574029.788037:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574029.788055:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574029.788064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574029/real 1788574029] req@ffff920a529b0700 x1875444569787904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574029.788072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569787904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574029.788078:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574029.788080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574029.788082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569787904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574029.788085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574029.788096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574029.788102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574029/real 1788574029] req@ffff920a529b0000 x1875444569788032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574029.788109:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569788032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574029.788114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574029.788116:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574029.788118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574029.788121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574029.788132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574029.788138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574029/real 1788574029] req@ffff920a529b1f80 x1875444569788160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574029.788146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569788160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574029.788151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574029.788153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574029.788155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574029.788158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574029.788168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574029.788173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574029/real 1788574029] req@ffff920a529b3100 x1875444569788288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574029.788179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569788288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574029.788184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574029.788186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574029.788188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574030.235787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574030.811735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574031.259765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574031.835715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574032.283783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574032.859713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574033.307807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574033.883732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574034.331768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574034.907673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574034.907713:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574034.907721:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574034.907725:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574034.907735:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10788 00000100:00080000:1.0:1788574034.907747:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574034.907754:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574034.907794:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569788416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574034.907825:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574034.907830:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574034.907832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574034.907838:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10788 00000100:00080000:1.0:1788574034.907843:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574034.907860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569788416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574034.907860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574034.907872:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574034.907890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569788544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574034.907903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574034.907908:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574034.907910:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574034.907915:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10788 00000100:00080000:1.0:1788574034.907920:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574034.907924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574034.907930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574034.907934:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574034.907947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569788672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574034.907958:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574034.907963:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574034.907966:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574034.907972:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574034/real 1788574034] req@ffff920a52891180 x1875444569788416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574034.907973:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10788 00000100:00080000:1.0:1788574034.907982:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574034.907990:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574034.907995:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569788416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788574034.907996:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574034.908015:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788574034.908017:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569788800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574034.908021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574034.908028:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569788416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574034.908041:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574034.908045:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574034.908047:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574034.908054:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10788 00000100:00100000:0.0:1788574034.908055:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788574034.908062:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574034.908070:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574034.908077:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574034.908082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574034.908098:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574034/real 1788574034] req@ffff920a52890e00 x1875444569788544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574034.908098:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569788928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574034.908135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569788544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574034.908142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574034.908146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574034.908148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574034.908153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574034.908167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574034.908174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574034/real 1788574034] req@ffff920a52890380 x1875444569788672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574034.908181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569788672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574034.908186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574034.908188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574034.908190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574034.908194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574034.908205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574034.908211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574034/real 1788574034] req@ffff920a52893480 x1875444569788800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574034.908219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569788800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574034.908224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574034.908226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574034.908227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574034.908238:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574034.908249:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574034.908255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574034/real 1788574034] req@ffff920a52893800 x1875444569788928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574034.908261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569788928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574034.908266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574034.908268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574034.908270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569788928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574035.355768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574035.931741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574036.379759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574036.955672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574037.403756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574037.979701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574038.427767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574039.003673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574039.451723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574040.027711:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574040.027722:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574040.027726:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574040.027736:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10793 00000100:00100000:0.0:1788574040.027746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574040.027747:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.027756:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574040.027789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569789056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574040.027814:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574040.027817:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574040.027818:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574040.027821:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10793 00000100:00080000:1.0:1788574040.027823:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.027826:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574040.027835:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574040.027844:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569789184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574040.027852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574040.027857:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574040.027859:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569789056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574040.027860:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574040.027875:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10793 00000100:00080000:1.0:1788574040.027878:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.027880:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574040.027883:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574040.027895:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569789312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574040.027901:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574040.027904:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574040.027905:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574040.027908:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10793 00000100:00080000:1.0:1788574040.027911:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.027914:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574040.027918:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574040.027926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569789440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574040.027931:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574040.027933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574040.027934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574040.027936:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10793 00000100:00080000:1.0:1788574040.027940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574040.027942:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574040.027945:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574040.027953:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569789568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574040.027960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574040.028008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff920a52892a00 x1875444569789056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574040.028023:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569789056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574040.028032:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574040.028035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574040.028039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569789056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574040.028061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574040.028077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574040.028085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff920a52891f80 x1875444569789184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574040.028098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569789184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574040.028107:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574040.028111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574040.028114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574040.028121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574040.028146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574040.028158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff920a52891500 x1875444569789312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574040.028171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569789312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574040.028180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574040.028184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574040.028188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574040.028195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574040.028218:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574040.028230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff920a52892680 x1875444569789440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574040.028243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569789440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574040.028252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574040.028256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574040.028260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574040.028266:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574040.028305:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574040.028318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574040/real 1788574040] req@ffff920a52890a80 x1875444569789568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574040.028339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569789568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574040.028349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574040.028352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574040.028356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574040.475770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574041.051675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574041.499741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574042.075676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574042.523739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574043.099664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574043.547756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574044.123740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574044.571807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574045.147724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574045.147763:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574045.147769:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574045.147773:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574045.147784:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10798 00000100:00080000:3.0:1788574045.147795:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574045.147802:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574045.147847:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569789696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574045.147873:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574045.147876:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574045.147877:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574045.147880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10798 00000100:00080000:3.0:1788574045.147882:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574045.147885:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574045.147895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574045.147908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569789824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574045.147910:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569789696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574045.147924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574045.147927:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574045.147928:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574045.147931:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10798 00000100:00080000:3.0:1788574045.147934:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574045.147937:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574045.147941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574045.147948:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569789952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574045.147955:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574045.147957:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574045.147959:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574045.147961:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10798 00000100:00080000:3.0:1788574045.147964:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574045.147966:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574045.147969:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574045.147976:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569790080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574045.147982:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574045.147984:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574045.147985:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574045.147987:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10798 00000100:00080000:3.0:1788574045.147990:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574045.147996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574045.147996:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574045.148002:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574045.148010:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569790208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574045.148049:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff920a529b1c00 x1875444569789696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574045.148067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569789696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574045.148075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574045.148078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574045.148081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569789696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574045.148104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574045.148123:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574045.148132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff920a529b3480 x1875444569789824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574045.148140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569789824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574045.148146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574045.148148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574045.148150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574045.148153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574045.148166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574045.148172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff920a529b2a00 x1875444569789952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574045.148179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569789952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574045.148184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574045.148186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574045.148188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569789952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574045.148191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574045.148202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574045.148208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff920a529b2d80 x1875444569790080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574045.148220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569790080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574045.148225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574045.148227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574045.148229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574045.148232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574045.148269:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574045.148277:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574045/real 1788574045] req@ffff920a529b3100 x1875444569790208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574045.148284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569790208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574045.148289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574045.148292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574045.148294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574045.595864:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574046.171701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574046.619731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574047.195707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574047.643722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574048.219706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574048.667787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574049.243742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574049.691784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574050.267702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574050.267709:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574050.267716:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574050.267722:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574050.267731:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10803 00000100:00080000:3.0:1788574050.267741:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574050.267750:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574050.267791:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569790336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574050.267823:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574050.267827:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574050.267828:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574050.267833:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10803 00000100:00080000:3.0:1788574050.267837:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574050.267840:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574050.267849:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574050.267861:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569790464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574050.267868:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574050.267871:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574050.267874:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574050.267877:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10803 00000100:00080000:3.0:1788574050.267879:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574050.267885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569790336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574050.267885:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574050.267894:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574050.267902:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569790592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574050.267908:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574050.267910:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574050.267911:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574050.267914:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10803 00000100:00080000:3.0:1788574050.267916:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574050.267918:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574050.267921:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574050.267929:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569790720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574050.267935:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574050.267937:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574050.267939:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574050.267941:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10803 00000100:00080000:3.0:1788574050.267944:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574050.267946:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574050.267950:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574050.267957:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569790848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574050.267970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574050.268010:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff920a529b1f80 x1875444569790336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574050.268036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569790336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574050.268048:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574050.268052:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574050.268056:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569790336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574050.268065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574050.268091:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574050.268104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff920a529b0000 x1875444569790464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574050.268114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569790464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574050.268120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574050.268123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574050.268125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574050.268145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574050.268173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574050.268185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff920a529b0700 x1875444569790592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574050.268229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569790592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574050.268247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574050.268251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574050.268254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574050.268259:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574050.268276:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574050.268284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff920a529b2300 x1875444569790720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574050.268300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569790720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574050.268306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574050.268309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574050.268311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574050.268314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574050.268326:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574050.268333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574050/real 1788574050] req@ffff920a529b0e00 x1875444569790848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574050.268340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569790848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574050.268345:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574050.268347:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574050.268349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569790848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574050.715763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574051.291727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574051.739806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574052.315712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574052.763789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574053.339831:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574053.787799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574054.363717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574054.811742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574055.387676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574055.387680:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.387688:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574055.387704:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.387719:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10808 00000100:00080000:3.0:1788574055.387730:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.387738:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574055.387776:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569790976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.387804:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.387808:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.387811:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.387816:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10808 00000100:00080000:3.0:1788574055.387820:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.387826:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574055.387836:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574055.387841:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569790976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574055.387855:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569791104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574055.387869:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.387874:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574055.387875:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.387880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10808 00000100:00080000:3.0:1788574055.387885:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.387891:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574055.387897:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574055.387901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574055.387914:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569791232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574055.387932:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff920a529b1180 x1875444569790976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574055.387933:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.387942:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574055.387946:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574055.387953:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10808 00000100:00100000:0.0:1788574055.387956:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569790976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574055.387959:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574055.387969:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788574055.387970:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574055.387976:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:3.0:1788574055.387977:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574055.387980:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569790976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574055.387995:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569791360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574055.388005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791104:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574055.388010:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574055.388016:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574055.388019:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574055.388025:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574055.388025:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10808 00000100:00080000:3.0:1788574055.388034:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574055.388040:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff920a529b0380 x1875444569791104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574055.388041:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574055.388050:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574055.388055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569791104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574055.388066:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569791488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574055.388069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574055.388075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574055.388082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574055.388091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574055.388105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574055.388113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff920a529b1500 x1875444569791232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574055.388121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569791232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574055.388126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574055.388129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574055.388131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574055.388140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574055.388151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574055.388158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff920a529b3b80 x1875444569791360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574055.388191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569791360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574055.388198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574055.388200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574055.388204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574055.388212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574055.388239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574055.388253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574055/real 1788574055] req@ffff920a529b1180 x1875444569791488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574055.388267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569791488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574055.388273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574055.388276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574055.388278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574055.835748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574056.411662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574056.859747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574057.435706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574057.883772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574058.459669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574058.907750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574059.483688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574059.931784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574060.507657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574060.507695:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574060.507701:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574060.507704:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574060.507712:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10813 00000100:00080000:3.0:1788574060.507721:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574060.507727:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574060.507761:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569791616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574060.507785:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574060.507787:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574060.507789:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574060.507792:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10813 00000100:00080000:3.0:1788574060.507794:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574060.507797:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574060.507807:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574060.507817:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569791616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574060.507821:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569791744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574060.507830:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574060.507833:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574060.507843:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574060.507847:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10813 00000100:00080000:3.0:1788574060.507849:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574060.507852:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574060.507856:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574060.507865:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569791872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574060.507872:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574060.507875:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574060.507876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574060.507878:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10813 00000100:00080000:3.0:1788574060.507881:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574060.507885:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574060.507887:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574060.507893:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574060.507900:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569792000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574060.507907:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574060.507910:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574060.507911:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574060.507913:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10813 00000100:00080000:3.0:1788574060.507916:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574060.507919:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574060.507922:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574060.507935:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569792128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574060.507936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574060] req@ffff920a529b0e00 x1875444569791616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574060.507967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569791616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574060.507976:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574060.507980:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574060.507983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569791616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574060.508005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574060.508025:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574060.508034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574060] req@ffff920a529b2300 x1875444569791744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574060.508042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569791744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574060.508047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574060.508049:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574060.508051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574060.508055:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574060.508078:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574060.508090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574060] req@ffff920a529b0700 x1875444569791872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574060.508101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569791872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574060.508109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574060.508113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574060.508116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569791872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574060.508136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574060.508156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574060.508172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574060] req@ffff920a529b0000 x1875444569792000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574060.508181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569792000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574060.508189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574060.508192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574060.508194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574060.508198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574060.508210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574060.508217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574060/real 1788574060] req@ffff920a529b1f80 x1875444569792128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574060.508224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569792128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574060.508228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574060.508230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574060.508232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574060.955754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574061.531748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574061.979763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574062.555674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574063.003733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574063.579696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574064.027755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574064.603699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574065.051797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574065.627730:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574065.627743:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1788574065.627745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574065.627748:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574065.627768:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10819 00000100:00080000:0.0:1788574065.627780:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574065.627789:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574065.627835:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569792256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574065.627869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574065.627874:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574065.627877:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574065.627883:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10819 00000100:00080000:0.0:1788574065.627888:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574065.627893:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574065.627905:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569792256:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574065.627906:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574065.627932:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569792384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574065.627943:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574065.627947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574065.627949:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574065.627954:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10819 00000100:00080000:0.0:1788574065.627969:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574065.627972:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574065.627978:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574065.627983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574065.627991:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569792512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574065.628002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574065.628005:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574065.628006:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574065.628011:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10819 00000100:00080000:0.0:1788574065.628018:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574065.628023:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574065.628028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574065/real 1788574065] req@ffff920a48350380 x1875444569792256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574065.628030:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574065.628062:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569792640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574065.628062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569792256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574065.628103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574065.628110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788574065.628114:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574065.628115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569792256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574065.628131:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574065.628134:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574065.628141:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10819 00000100:00100000:1.0:1788574065.628151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792384:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574065.628156:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574065.628163:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574065.628169:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574065.628178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574065.628186:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569792768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574065.628195:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574065/real 1788574065] req@ffff920a48350700 x1875444569792384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574065.628208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569792384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574065.628215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574065.628219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574065.628223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574065.628230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574065.628243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574065.628250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574065/real 1788574065] req@ffff920a48353480 x1875444569792512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574065.628257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569792512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574065.628263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574065.628265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574065.628267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574065.628271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574065.628285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574065.628295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574065/real 1788574065] req@ffff920a48351500 x1875444569792640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574065.628309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569792640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574065.628317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574065.628320:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574065.628324:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574065.628343:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574065.628359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574065.628368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574065/real 1788574065] req@ffff920a48350a80 x1875444569792768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574065.628375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569792768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574065.628381:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574065.628383:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574065.628385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569792768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574066.075818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574066.651656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574067.099822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574067.675705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574068.123776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574068.699733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574069.147796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574069.723688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574070.171818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574070.747684:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574070.747695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788574070.747698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574070.747699:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574070.747714:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10824 00000100:00080000:2.0:1788574070.747726:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574070.747735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574070.747781:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569792896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574070.747807:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574070.747811:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574070.747812:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574070.747816:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10824 00000100:00080000:2.0:1788574070.747818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574070.747821:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574070.747831:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574070.747841:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569793024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574070.747848:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569792896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574070.747852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574070.747857:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574070.747858:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574070.747861:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10824 00000100:00080000:2.0:1788574070.747863:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574070.747866:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574070.747868:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574070.747877:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569793152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574070.747883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574070.747886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574070.747888:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574070.747891:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10824 00000100:00080000:2.0:1788574070.747894:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574070.747897:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574070.747900:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574070.747907:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569793280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574070.747913:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574070.747916:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574070.747917:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574070.747920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10824 00000100:00080000:2.0:1788574070.747922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574070.747928:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788574070.747930:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574070.747932:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574070.747940:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569793408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574070.747976:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574070/real 1788574070] req@ffff920a60ac4a80 x1875444569792896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574070.747998:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569792896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574070.748019:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574070.748023:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574070.748027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569792896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574070.748063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574070.748084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574070.748095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574070/real 1788574070] req@ffff920a60ac4700 x1875444569793024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574070.748104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569793024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574070.748110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574070.748113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574070.748115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574070.748120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574070.748131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574070.748137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574070/real 1788574070] req@ffff920a60ac4e00 x1875444569793152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574070.748145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569793152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574070.748150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574070.748152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574070.748154:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574070.748158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574070.748168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574070.748174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574070/real 1788574070] req@ffff920a60ac7800 x1875444569793280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574070.748181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569793280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574070.748186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574070.748188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574070.748190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574070.748193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574070.748204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574070.748210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574070/real 1788574070] req@ffff920a60ac5c00 x1875444569793408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574070.748216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569793408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574070.748221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574070.748223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574070.748225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574071.195787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574071.771703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574072.219804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574072.795714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574073.243777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574073.819693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574074.267757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574074.843702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574075.291840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574075.867706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574075.867725:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574075.867734:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574075.867739:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574075.867752:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10829 00000100:00080000:2.0:1788574075.867765:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574075.867774:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574075.867825:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569793536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574075.867863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574075.867867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574075.867869:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574075.867874:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10829 00000100:00080000:2.0:1788574075.867878:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574075.867884:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574075.867896:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574075.867911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569793536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574075.867915:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569793664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574075.867928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574075.867932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574075.867935:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574075.867939:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10829 00000100:00080000:2.0:1788574075.867943:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574075.867948:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574075.867953:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574075.867967:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569793792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574075.867978:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574075.867982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574075.867984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574075.867989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10829 00000100:00080000:2.0:1788574075.867993:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574075.867998:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574075.868019:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574075.868030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574075.868036:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569793920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574075.868050:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574075.868054:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574075.868056:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574075.868061:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10829 00000100:00100000:1.0:1788574075.868066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793792:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788574075.868066:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574075.868077:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574075.868084:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574075.868091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574075.868100:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569794048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574075.868122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574076.315740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574076.891730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574077.339792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574077.915682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574078.363735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574078.939685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574078.939945:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574078.940002:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574078.940018:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574078.940073:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574078.940084:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574078.940093:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574078.940101:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788574078.940148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574075/real 1788574078] req@ffff920a60ac4e00 x1875444569794048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574078.940180:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569794048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574078.940195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574078.940202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574078.940207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794048:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574078.940216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574075/real 1788574078] req@ffff920a60ac7800 x1875444569793920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574078.940228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569793920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574078.940236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574078.940246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574078.940250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574078.940257:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574075/real 1788574078] req@ffff920a60ac4380 x1875444569793664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574078.940270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569793664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574078.940279:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574078.940283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574078.940286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574078.940294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574075/real 1788574078] req@ffff920a60ac5c00 x1875444569793792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574078.940306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569793792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574078.940314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574078.940317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574078.940320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569793792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574078.940327:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574075/real 1788574078] req@ffff920a60ac4000 x1875444569793536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574078.940339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569793536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574078.940346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574078.940350:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574078.940353:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569793536:192.168.202.157@tcp:250:kworker.0 00010000:00010000:0.0:1788574079.387755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574079.963651:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574080.411739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574080.987656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574080.987657:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574080.987667:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574080.987671:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574080.987682:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10834 00000100:00080000:2.0:1788574080.987692:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574080.987699:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574080.987740:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569794176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574080.987767:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574080.987769:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574080.987771:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574080.987774:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10834 00000100:00080000:2.0:1788574080.987777:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574080.987779:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574080.987789:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574080.987799:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569794304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574080.987806:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574080.987809:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574080.987810:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574080.987813:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10834 00000100:00080000:2.0:1788574080.987815:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574080.987817:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574080.987821:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574080.987826:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569794176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574080.987831:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569794432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574080.987841:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574080.987843:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574080.987844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574080.987846:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10834 00000100:00080000:2.0:1788574080.987848:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574080.987850:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574080.987855:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574080.987867:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569794560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574080.987876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574080.987880:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574080.987882:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574080.987886:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10834 00000100:00080000:2.0:1788574080.987890:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574080.987894:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574080.987899:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574080.987921:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574080.987925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569794688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574080.987980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574080/real 1788574080] req@ffff920a60ac4700 x1875444569794176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574080.988002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569794176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574080.988015:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574080.988020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574080.988025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569794176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574080.988033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574080.988059:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574080.988075:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574080/real 1788574080] req@ffff920a60ac4a80 x1875444569794304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574080.988087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569794304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574080.988095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574080.988098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574080.988102:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574080.988134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574080.988154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574080.988165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574080/real 1788574080] req@ffff920a60ac5500 x1875444569794432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574080.988178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569794432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574080.988191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574080.988195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574080.988198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574080.988203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574080.988225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574080.988235:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574080/real 1788574080] req@ffff920a60ac6300 x1875444569794560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574080.988245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569794560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574080.988253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574080.988255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574080.988258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574080.988264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574080.988283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574080.988293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574080/real 1788574080] req@ffff920a60ac7100 x1875444569794688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574080.988303:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569794688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574105 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574080.988311:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574080.988314:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574080.988318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574081.435756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574082.011676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574082.459804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574083.035714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574083.483749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574084.059748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574084.507769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574085.083674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574085.531788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574086.107685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574086.107700:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.107706:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574086.107711:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.107722:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10839 00000100:00080000:2.0:1788574086.107733:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.107742:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574086.107795:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569794816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.107820:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.107823:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574086.107824:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.107827:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10839 00000100:00080000:2.0:1788574086.107830:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.107832:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574086.107842:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574086.107851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569794944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.107859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.107862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574086.107865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574086.107866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569794816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574086.107870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10839 00000100:00080000:2.0:1788574086.107874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.107878:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574086.107881:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574086.107890:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569795072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.107897:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.107899:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574086.107900:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.107902:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10839 00000100:00080000:2.0:1788574086.107904:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.107906:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574086.107910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574086.107917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569795200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574086.107923:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574086.107936:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574086.107937:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574086.107940:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10839 00000100:00080000:2.0:1788574086.107943:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574086.107946:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574086.107950:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574086.107958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569795328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574086.107973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574086.108013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff920a60ac5180 x1875444569794816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.108032:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569794816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.108044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.108048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574086.108053:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569794816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574086.108081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574086.108103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574086.108116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff920a60ac6a00 x1875444569794944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.108130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569794944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.108139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.108143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574086.108147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569794944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574086.108156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574086.108175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574086.108186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff920a60ac7100 x1875444569795072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.108198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569795072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.108207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.108210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574086.108213:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574086.108219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574086.108237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574086.108246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff920a60ac6300 x1875444569795200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.108258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569795200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.108267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.108271:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574086.108275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574086.108280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574086.108297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574086.108306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574086/real 1788574086] req@ffff920a60ac5500 x1875444569795328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574086.108318:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569795328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574086.108326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574086.108330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574086.108333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574086.555882:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574087.131677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574087.579735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574088.155689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574088.603778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574089.179829:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574089.627773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574090.203692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574090.651700:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574091.227698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574091.227703:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574091.227712:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574091.227716:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574091.227728:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10844 00000100:00080000:0.0:1788574091.227739:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574091.227747:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574091.227789:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569795456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574091.227816:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574091.227820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574091.227821:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574091.227826:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10844 00000100:00080000:0.0:1788574091.227831:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574091.227837:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574091.227844:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569795456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574091.227850:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574091.227870:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569795584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574091.227880:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574091.227898:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574091.227901:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574091.227906:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10844 00000100:00080000:0.0:1788574091.227911:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574091.227915:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574091.227920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574091.227932:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569795712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574091.227940:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574091.227947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788574091.227947:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574091.227951:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574091.227957:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10844 00000100:00080000:0.0:1788574091.227961:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574091.227964:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574091.227969:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574091.227978:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569795840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574091.227985:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574091.227987:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574091.227988:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574091.227992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10844 00000100:00080000:0.0:1788574091.227995:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574091.227998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff920a48353b80 x1875444569795456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574091.228000:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574091.228005:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574091.228016:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569795968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574091.228034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569795456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574091.228045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574091.228049:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574091.228053:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569795456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574091.228076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574091.228098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574091.228109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff920a48353800 x1875444569795584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574091.228117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569795584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574091.228123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574091.228125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574091.228127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574091.228131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574091.228142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574091.228148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff920a48351c00 x1875444569795712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574091.228161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569795712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574091.228166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574091.228168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574091.228170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574091.228173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574091.228185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574091.228199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff920a48350a80 x1875444569795840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574091.228208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569795840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574091.228214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574091.228216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574091.228218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574091.228222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574091.228235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574091.228242:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574091/real 1788574091] req@ffff920a48351500 x1875444569795968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574091.228249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569795968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574091.228259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574091.228261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574091.228263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569795968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574091.675802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574092.251737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574092.699802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574093.275679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574093.723736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574094.299715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574094.747778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574095.323704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574095.771752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574096.347719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574096.347728:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574096.347735:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574096.347740:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574096.347752:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10849 00000100:00080000:2.0:1788574096.347762:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574096.347770:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574096.347806:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569796096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574096.347835:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574096.347839:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574096.347841:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574096.347860:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10849 00000100:00080000:2.0:1788574096.347865:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574096.347871:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574096.347876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569796096:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574096.347884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574096.347903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569796224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574096.347916:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574096.347921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574096.347924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574096.347929:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10849 00000100:00080000:2.0:1788574096.347934:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574096.347939:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788574096.347944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574096.347945:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574096.347961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569796352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574096.347972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574096.347976:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574096.347979:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574096.347983:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10849 00000100:00080000:2.0:1788574096.347986:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574096.347987:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff920a60ac4a80 x1875444569796096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574096.347991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574096.348000:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574096.348011:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569796096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574096.348012:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569796480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574096.348041:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574096.348042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574096.348046:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574096.348049:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574096.348050:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788574096.348054:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10849 00000100:00100000:1.0:1788574096.348057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569796096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574096.348058:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574096.348066:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574096.348070:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574096.348076:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569796608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574096.348082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574096.348104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574096.348113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff920a60ac4700 x1875444569796224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.348121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569796224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.348127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.348129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574096.348131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574096.348135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574096.348153:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574096.348160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff920a60ac4000 x1875444569796352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.348167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569796352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.348173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.348175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574096.348177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574096.348180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574096.348191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574096.348196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff920a60ac5c00 x1875444569796480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.348204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569796480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.348209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.348210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574096.348212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574096.348220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574096.348231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574096.348236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574096/real 1788574096] req@ffff920a60ac4380 x1875444569796608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574096.348245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569796608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574121 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574096.348252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574096.348256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574096.348259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574096.795731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574097.371720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574097.819747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574098.395712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574098.843789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574099.419677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574099.867750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574100.443696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574100.891783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574101.467722:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574101.467723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574101.467736:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574101.467742:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574101.467755:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10854 00000100:00080000:2.0:1788574101.467768:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574101.467780:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574101.467846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569796736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574101.467874:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574101.467877:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574101.467879:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574101.467883:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10854 00000100:00080000:2.0:1788574101.467886:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574101.467888:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574101.467898:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574101.467909:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569796864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574101.467918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574101.467921:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574101.467924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574101.467927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569796736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574101.467930:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10854 00000100:00080000:2.0:1788574101.467936:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574101.467939:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574101.467949:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574101.467961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569796992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574101.467969:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574101.467972:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574101.467973:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574101.467976:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10854 00000100:00080000:2.0:1788574101.467979:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574101.467982:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574101.467986:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574101.467994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569797120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574101.468001:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574101.468003:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574101.468004:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574101.468009:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10854 00000100:00080000:2.0:1788574101.468012:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574101.468014:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574101.468015:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574101.468022:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574101.468030:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569797248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574101.468058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff920a60ac7800 x1875444569796736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.468074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569796736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.468083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.468087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574101.468090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569796736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574101.468114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574101.468139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574101.468154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff920a60ac4e00 x1875444569796864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.468168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569796864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.468178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.468182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574101.468186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574101.468193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574101.468210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574101.468219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff920a60ac4380 x1875444569796992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.468228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569796992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.468239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.468242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574101.468244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569796992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574101.468248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574101.468260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574101.468266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff920a60ac5c00 x1875444569797120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.468274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569797120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.468280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.468281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574101.468283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574101.468287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574101.468297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574101.468303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574101/real 1788574101] req@ffff920a60ac4000 x1875444569797248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574101.468310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569797248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574126 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574101.468315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574101.468318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574101.468320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574101.915777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574102.491696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574102.939735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574103.515671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574103.963812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574104.539685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574104.987719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574105.563776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574106.011853:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574106.587652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574106.587666:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574106.587672:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574106.587676:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574106.587686:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10859 00000100:00080000:2.0:1788574106.587713:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574106.587722:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574106.587762:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569797376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574106.587792:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574106.587797:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574106.587799:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574106.587803:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10859 00000100:00080000:2.0:1788574106.587807:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574106.587812:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574106.587823:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574106.587836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569797376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574106.587840:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569797504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574106.587851:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574106.587853:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574106.587855:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574106.587859:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10859 00000100:00080000:2.0:1788574106.587862:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574106.587865:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574106.587870:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574106.587880:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569797632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574106.587887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574106.587890:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574106.587891:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574106.587893:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10859 00000100:00080000:2.0:1788574106.587896:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574106.587900:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574106.587903:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574106.587908:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574106.587922:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569797760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574106.587932:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574106.587935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff920a60ac4700 x1875444569797376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574106.587938:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574106.587943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574106.587952:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10859 00000100:00080000:2.0:1788574106.587957:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574106.587958:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569797376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574106.587967:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788574106.587974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788574106.587975:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574106.587982:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574106.587986:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569797376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574106.587990:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569797888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574106.588009:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574106.588029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574106.588039:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff920a60ac4a80 x1875444569797504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.588047:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569797504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.588053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.588056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574106.588058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574106.588064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574106.588075:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574106.588094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff920a60ac5500 x1875444569797632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.588103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569797632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.588108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.588111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574106.588113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574106.588116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574106.588129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574106.588135:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff920a60ac6300 x1875444569797760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.588142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569797760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.588147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.588149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574106.588151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574106.588155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574106.588165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574106.588171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574106/real 1788574106] req@ffff920a60ac7100 x1875444569797888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574106.588179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569797888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574131 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574106.588185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574106.588186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574106.588188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569797888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574107.035788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574107.611704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574108.059764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574108.635693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574109.083820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574109.659675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574110.107802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574110.683679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574111.131707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574111.707681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574111.707688:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574111.707694:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574111.707698:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574111.707707:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10865 00000100:00080000:1.0:1788574111.707717:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574111.707722:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574111.707754:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569798016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574111.707780:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574111.707783:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574111.707785:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574111.707788:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10865 00000100:00080000:1.0:1788574111.707791:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574111.707794:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574111.707802:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574111.707812:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569798144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574111.707818:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569798016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574111.707834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574111.707836:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574111.707838:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574111.707843:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10865 00000100:00080000:1.0:1788574111.707847:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574111.707850:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574111.707856:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574111.707869:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569798272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574111.707879:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574111.707885:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574111.707887:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574111.707890:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574111.707895:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10865 00000100:00080000:1.0:1788574111.707899:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574111.707905:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574111.707911:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574111.707924:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569798400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574111.707929:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574111/real 1788574111] req@ffff920a52893b80 x1875444569798016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574111.707949:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574111.707954:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574111.707957:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569798016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574111.707958:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574111.707968:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10865 00000100:00080000:2.0:1788574111.707974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574111.707975:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574111.707984:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788574111.707985:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574111.707991:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569798016:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574111.707995:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574111.708009:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569798528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574111.708014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574111.708038:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574111.708050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574111/real 1788574111] req@ffff920a52890000 x1875444569798144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574111.708059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569798144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574111.708065:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574111.708068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574111.708070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574111.708075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574111.708085:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574111.708091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574111/real 1788574111] req@ffff920a52890a80 x1875444569798272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574111.708098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569798272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574111.708104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574111.708105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574111.708107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574111.708110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574111.708120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574111.708126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574111/real 1788574111] req@ffff920a52892680 x1875444569798400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574111.708133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569798400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574111.708141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574111.708144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574111.708148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574111.708164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574111.708183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574111.708192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574111/real 1788574111] req@ffff920a52891500 x1875444569798528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574111.708200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569798528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574111.708205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574111.708207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574111.708209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574112.155760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574112.731684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574113.179770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574113.755724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574114.203730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574114.779669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574115.227759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574115.803710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574116.251779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574116.827674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574116.827711:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574116.827719:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574116.827725:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574116.827737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10870 00000100:00080000:2.0:1788574116.827751:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574116.827759:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574116.827803:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569798656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574116.827829:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574116.827832:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574116.827834:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574116.827837:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10870 00000100:00080000:2.0:1788574116.827840:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574116.827843:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574116.827853:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574116.827866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569798656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574116.827866:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569798784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574116.827880:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574116.827883:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574116.827884:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574116.827887:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10870 00000100:00080000:2.0:1788574116.827888:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574116.827891:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574116.827894:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574116.827903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569798912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574116.827910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574116.827913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574116.827914:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574116.827916:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10870 00000100:00080000:2.0:1788574116.827918:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574116.827921:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574116.827924:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574116.827931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569799040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574116.827937:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574116.827939:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574116.827940:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574116.827944:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10870 00000100:00080000:2.0:1788574116.827946:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574116.827949:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574116.827952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574116.827958:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574116.827960:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569799168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574116.828007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574116/real 1788574116] req@ffff920a60ac6a00 x1875444569798656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574116.828025:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569798656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574116.828035:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574116.828038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574116.828041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569798656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574116.828065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574116.828083:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574116.828092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574116/real 1788574116] req@ffff920a60ac5180 x1875444569798784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574116.828101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569798784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574116.828106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574116.828108:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574116.828111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574116.828116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574116.828127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574116.828141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574116/real 1788574116] req@ffff920a60ac7100 x1875444569798912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574116.828149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569798912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574116.828155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574116.828157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574116.828159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569798912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574116.828162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574116.828175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574116.828181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574116/real 1788574116] req@ffff920a60ac6300 x1875444569799040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574116.828188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569799040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574116.828194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574116.828195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574116.828197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574116.828201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574116.828211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574116.828224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574116/real 1788574116] req@ffff920a60ac5500 x1875444569799168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574116.828232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569799168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574116.828238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574116.828239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574116.828241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574117.275767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574117.851735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574118.299760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574118.875665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574119.323744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574119.899746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574120.347712:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574120.923654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574121.371764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574121.947661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574121.947673:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574121.947679:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574121.947682:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574121.947692:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10875 00000100:00080000:2.0:1788574121.947711:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574121.947720:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574121.947758:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569799296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574121.947780:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574121.947783:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574121.947785:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574121.947788:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10875 00000100:00080000:2.0:1788574121.947792:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574121.947795:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574121.947804:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574121.947810:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569799296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574121.947818:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569799424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574121.947828:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574121.947830:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574121.947831:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574121.947834:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10875 00000100:00080000:2.0:1788574121.947836:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574121.947839:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574121.947842:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574121.947850:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569799552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574121.947856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574121.947859:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574121.947860:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574121.947862:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10875 00000100:00080000:2.0:1788574121.947865:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574121.947869:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574121.947870:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574121.947876:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574121.947885:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569799680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574121.947892:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574121.947894:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574121.947896:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574121.947900:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10875 00000100:00100000:1.0:1788574121.947904:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574121/real 1788574121] req@ffff920a60ac4a80 x1875444569799296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574121.947904:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574121.947910:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574121.947916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574121.947925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569799808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574121.947929:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569799296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574121.947942:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574121.947946:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574121.947949:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569799296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574121.947968:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574121.947994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574121.948003:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574121/real 1788574121] req@ffff920a60ac4700 x1875444569799424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574121.948016:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569799424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574121.948022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574121.948025:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574121.948027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574121.948030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574121.948042:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574121.948048:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574121/real 1788574121] req@ffff920a60ac4000 x1875444569799552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574121.948055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569799552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574121.948060:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574121.948062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574121.948064:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574121.948067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574121.948078:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574121.948087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574121/real 1788574121] req@ffff920a60ac5c00 x1875444569799680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574121.948098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569799680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574121.948113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574121.948117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574121.948120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574121.948127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574121.948147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574121.948155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574121/real 1788574121] req@ffff920a60ac4380 x1875444569799808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574121.948163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569799808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574146 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574121.948169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574121.948171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574121.948173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569799808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574122.395715:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574122.971736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574123.419787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574123.995659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574124.443719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574125.019744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574125.467780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574126.043660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574126.491736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574127.067693:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574127.067708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1788574127.067709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574127.067714:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574127.067730:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10880 00000100:00080000:0.0:1788574127.067742:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574127.067752:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574127.067799:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569799936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574127.067836:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574127.067840:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574127.067842:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574127.067847:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10880 00000100:00080000:0.0:1788574127.067850:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574127.067855:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574127.067866:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574127.067879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569799936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574127.067882:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569800064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574127.067895:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574127.067898:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574127.067900:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574127.067905:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10880 00000100:00080000:0.0:1788574127.067908:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574127.067911:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574127.067916:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574127.067925:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569800192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574127.067934:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574127.067937:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574127.067939:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574127.067942:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10880 00000100:00080000:0.0:1788574127.067946:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574127.067950:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574127.067965:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574127.067974:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569800320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574127.067980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574127.067986:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574127.067990:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574127.067991:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574127.067994:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10880 00000100:00080000:0.0:1788574127.067997:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574127.068000:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574127.068003:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574127.068010:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569800448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574127.068024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574127] req@ffff920a48353480 x1875444569799936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574127.068046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569799936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574127.068058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574127.068063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574127.068067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569799936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574127.068128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574127.068151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574127.068163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574127] req@ffff920a48350700 x1875444569800064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574127.068172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569800064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574127.068178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574127.068181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574127.068184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574127.068188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574127.068201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574127.068207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574127] req@ffff920a48350380 x1875444569800192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574127.068215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569800192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574127.068220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574127.068222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574127.068224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574127.068228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574127.068240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574127.068247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574127] req@ffff920a48351500 x1875444569800320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574127.068256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569800320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574127.068262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574127.068265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574127.068267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574127.068271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574127.068283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574127.068290:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574127/real 1788574127] req@ffff920a48350a80 x1875444569800448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574127.068298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569800448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574152 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574127.068303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574127.068305:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574127.068307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574127.515754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574128.091683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574128.539783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574129.115695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574129.563782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574130.139711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574130.587732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574131.163691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574131.611752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574132.187681:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574132.187681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574132.187690:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574132.187696:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574132.187705:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10885 00000100:00080000:0.0:1788574132.187713:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574132.187719:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574132.187751:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569800576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574132.187777:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574132.187780:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574132.187781:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574132.187784:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10885 00000100:00080000:0.0:1788574132.187786:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574132.187789:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574132.187796:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574132.187805:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569800704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574132.187813:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574132.187817:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569800576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574132.187818:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574132.187820:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574132.187828:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10885 00000100:00080000:0.0:1788574132.187831:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574132.187833:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574132.187836:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574132.187843:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569800832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574132.187850:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574132.187852:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574132.187854:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574132.187857:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10885 00000100:00080000:0.0:1788574132.187860:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574132.187863:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574132.187866:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574132.187873:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569800960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574132.187879:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574132.187881:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574132.187883:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574132.187888:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574132.187891:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10885 00000100:00080000:0.0:1788574132.187895:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574132.187898:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574132.187901:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574132.187909:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569801088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574132.187942:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff920a48351c00 x1875444569800576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.187959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569800576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.187972:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.187975:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574132.187978:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569800576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574132.188002:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.188021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574132.188031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff920a48353800 x1875444569800704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.188039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569800704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.188044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.188047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574132.188049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574132.188053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574132.188064:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574132.188077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff920a48353b80 x1875444569800832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.188084:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569800832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.188090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.188092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574132.188094:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574132.188098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574132.188115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574132.188122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff920a48350a80 x1875444569800960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.188130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569800960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.188135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.188138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574132.188140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569800960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574132.188143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574132.188153:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574132.188159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574132/real 1788574132] req@ffff920a48351500 x1875444569801088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574132.188166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569801088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574157 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574132.188171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574132.188173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574132.188175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574132.635727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574133.211693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574133.659795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574134.235684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574134.683747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574135.259690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574135.707734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574136.283677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574136.731744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574137.307697:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574137.307708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574137.307708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574137.307713:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574137.307725:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10890 00000100:00080000:1.0:1788574137.307736:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574137.307744:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574137.307814:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569801216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574137.307838:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574137.307841:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574137.307843:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574137.307846:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10890 00000100:00080000:1.0:1788574137.307849:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574137.307852:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574137.307863:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574137.307872:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569801344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574137.307880:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574137.307883:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574137.307885:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574137.307890:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10890 00000100:00100000:2.0:1788574137.307890:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569801216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574137.307896:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574137.307900:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574137.307903:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574137.307911:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569801472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574137.307922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574137.307925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574137.307927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574137.307930:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10890 00000100:00080000:1.0:1788574137.307932:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574137.307934:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574137.307938:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574137.307945:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569801600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574137.307951:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574137.307953:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574137.307955:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574137.307958:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10890 00000100:00080000:1.0:1788574137.307960:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574137.307962:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574137.307965:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574137.307972:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569801728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574137.307977:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574137.308024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff920a52891f80 x1875444569801216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574137.308040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569801216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574137.308050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574137.308053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574137.308056:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569801216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574137.308079:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574137.308098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574137.308107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff920a52892a00 x1875444569801344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574137.308115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569801344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574137.308121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574137.308124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574137.308126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574137.308129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574137.308141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574137.308146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff920a52893800 x1875444569801472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574137.308154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569801472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574137.308160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574137.308161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574137.308163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574137.308168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574137.308179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574137.308184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff920a52893480 x1875444569801600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574137.308191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569801600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574137.308196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574137.308198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574137.308200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574137.308203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574137.308213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574137.308218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574137/real 1788574137] req@ffff920a52890380 x1875444569801728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574137.308225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569801728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574162 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574137.308230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574137.308232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574137.308233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574137.755779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574138.331694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574138.779760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574139.355734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574139.803729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574140.379719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574140.827814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574141.403677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574141.851784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574142.427739:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574142.427741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574142.427751:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574142.427756:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574142.427769:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10895 00000100:00080000:1.0:1788574142.427783:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574142.427791:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574142.427838:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569801856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574142.427864:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574142.427867:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574142.427868:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574142.427871:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10895 00000100:00080000:1.0:1788574142.427873:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574142.427875:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574142.427885:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574142.427895:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569801984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574142.427902:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574142.427906:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574142.427907:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574142.427911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569801856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574142.427913:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10895 00000100:00080000:1.0:1788574142.427915:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574142.427918:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574142.427924:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574142.427933:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569802112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574142.427939:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574142.427942:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574142.427943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574142.427946:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10895 00000100:00080000:1.0:1788574142.427948:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574142.427951:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574142.427954:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574142.427962:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569802240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574142.427968:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574142.427970:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574142.427972:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574142.427974:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10895 00000100:00080000:1.0:1788574142.427976:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574142.427979:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574142.427982:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574142.427988:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569802368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574142.428036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574142.428071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574142.428088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574142.428103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574142.875780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574143.451692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574143.899761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574144.475699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574144.923758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574145.499680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574145.499866:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574145.499909:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574145.499925:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574145.499959:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574145.499965:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574145.499972:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574145.499977:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788574145.500032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574145] req@ffff920a52891180 x1875444569801984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574145.500071:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569801984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574145.500085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574145.500092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574145.500098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569801984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574145.500113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574145] req@ffff920a52892300 x1875444569802112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574145.500127:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569802112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574145.500137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574145.500141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574145.500145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574145.500153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574145] req@ffff920a52892d80 x1875444569802240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574145.500168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569802240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574145.500178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574145.500181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574145.500185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574145.500193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574145] req@ffff920a52891880 x1875444569802368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574145.500205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569802368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574145.500213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574145.500217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574145.500221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802368:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574145.500230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574142/real 1788574145] req@ffff920a52890e00 x1875444569801856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574145.500245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569801856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574167 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574145.500254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574145.500258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574145.500262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569801856:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788574145.947756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574146.523689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574146.971783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574147.547663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574147.547715:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574147.547727:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574147.547732:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574147.547743:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10900 00000100:00080000:2.0:1788574147.547754:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574147.547775:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574147.547813:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569802496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574147.547844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574147.547848:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574147.547851:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574147.547857:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10900 00000100:00080000:2.0:1788574147.547862:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574147.547869:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574147.547881:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574147.547898:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569802496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574147.547912:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569802624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574147.547929:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574147.547934:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574147.547936:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574147.547942:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10900 00000100:00080000:2.0:1788574147.547947:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574147.547953:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574147.547960:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574147.547979:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569802752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574147.547991:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788574147.547995:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574147.548000:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574147.548005:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574147.548013:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10900 00000100:00080000:2.0:1788574147.548021:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574147.548027:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574147.548034:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574147.548052:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569802880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574147.548063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff920a60ac4e00 x1875444569802496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574147.548075:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574147.548094:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574147.548097:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574147.548106:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10900 00000100:00080000:2.0:1788574147.548111:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574147.548112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569802496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574147.548121:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574147.548142:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574147.548148:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.548157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574147.548163:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569803008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574147.548171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569802496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574147.548233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574147.548261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574147.548276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff920a60ac7800 x1875444569802624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.548303:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569802624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.548313:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.548316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574147.548319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574147.548326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574147.548346:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574147.548357:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff920a60ac4380 x1875444569802752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.548370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569802752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.548378:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.548382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574147.548385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574147.548392:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574147.548413:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574147.548423:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff920a60ac5c00 x1875444569802880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.548435:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569802880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.548481:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.548485:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574147.548489:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569802880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574147.548496:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574147.548516:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574147.548528:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574147/real 1788574147] req@ffff920a60ac4000 x1875444569803008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574147.548540:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569803008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574147.548548:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574147.548552:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574147.548555:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574147.995761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574148.571698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574149.019814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574149.595707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574150.043818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574150.619678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574151.067767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574151.643664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574152.091798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574152.667695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574152.667708:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574152.667714:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574152.667723:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574152.667732:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10906 00000100:00080000:1.0:1788574152.667742:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574152.667754:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574152.667791:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569803136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574152.667823:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574152.667827:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574152.667828:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574152.667831:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10906 00000100:00080000:1.0:1788574152.667834:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574152.667837:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574152.667846:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574152.667857:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569803264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574152.667868:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574152.667871:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569803136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574152.667872:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574152.667875:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574152.667888:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10906 00000100:00080000:1.0:1788574152.667891:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574152.667893:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574152.667897:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574152.667913:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569803392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574152.667923:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574152.667927:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574152.667929:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574152.667934:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10906 00000100:00080000:1.0:1788574152.667939:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574152.667943:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574152.667952:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574152.667964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574152.667967:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569803520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574152.667976:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574152.667979:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574152.667982:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574152.667986:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10906 00000100:00080000:1.0:1788574152.667991:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574152.667995:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574152.668001:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574152.668011:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574152/real 1788574152] req@ffff920a52890e00 x1875444569803136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574152.668021:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569803648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574152.668031:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569803136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.668050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.668055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574152.668058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569803136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574152.668085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574152.668104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574152.668114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574152/real 1788574152] req@ffff920a52891880 x1875444569803264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.668123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569803264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.668129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.668131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574152.668133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574152.668144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574152.668157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574152.668164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574152/real 1788574152] req@ffff920a52892d80 x1875444569803392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.668171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569803392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.668177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.668179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574152.668181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574152.668185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574152.668196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574152.668202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574152/real 1788574152] req@ffff920a52892300 x1875444569803520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.668210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569803520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.668215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.668217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574152.668218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574152.668222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574152.668247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574152.668259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574152/real 1788574152] req@ffff920a52891180 x1875444569803648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574152.668272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569803648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574152.668300:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574152.668304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574152.668307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574153.115749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574153.691658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574154.139746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574154.715709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574155.163764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574155.739698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574156.187889:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574156.763770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574157.211737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574157.787738:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574157.787744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574157.787748:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574157.787755:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574157.787764:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10911 00000100:00080000:3.0:1788574157.787774:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574157.787790:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574157.787828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569803776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574157.787863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574157.787866:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574157.787867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574157.787871:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10911 00000100:00080000:3.0:1788574157.787874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574157.787877:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574157.787893:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574157.787905:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569803904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574157.787912:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569803776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574157.787919:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574157.787923:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574157.787924:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574157.787928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10911 00000100:00080000:3.0:1788574157.787931:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574157.787934:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574157.787937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574157.787947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569804032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574157.787955:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574157.787958:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574157.787959:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574157.787961:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10911 00000100:00080000:3.0:1788574157.787967:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574157.787969:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574157.787973:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574157.787978:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574157.787985:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569804160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574157.787992:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574157.787994:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574157.787996:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574157.787998:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10911 00000100:00080000:3.0:1788574157.788001:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574157.788004:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574157.788007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574157.788014:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574157/real 1788574157] req@ffff920a529b3100 x1875444569803776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574157.788017:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569804288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574157.788058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569803776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574157.788067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574157.788079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574157.788082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569803776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574157.788114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574157.788172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574157.788188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574157/real 1788574157] req@ffff920a529b2d80 x1875444569803904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574157.788202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569803904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574157.788212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574157.788215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574157.788219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569803904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574157.788226:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574157.788249:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574157.788262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574157/real 1788574157] req@ffff920a529b2a00 x1875444569804032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574157.788279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569804032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574157.788294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574157.788297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574157.788301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574157.788307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574157.788340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574157.788350:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574157/real 1788574157] req@ffff920a529b3480 x1875444569804160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574157.788358:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569804160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574157.788363:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574157.788365:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574157.788367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574157.788371:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574157.788384:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574157.788390:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574157/real 1788574157] req@ffff920a529b1c00 x1875444569804288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574157.788397:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569804288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574157.788401:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574157.788403:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574157.788406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574158.235706:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574158.811688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574159.259756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574159.835694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574160.283824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574160.859720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574161.307807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574161.883718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574162.331799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574162.907677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574162.907726:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574162.907737:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574162.907743:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574162.907757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10916 00000100:00080000:2.0:1788574162.907771:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574162.907781:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574162.907826:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569804416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574162.907859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574162.907865:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574162.907868:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574162.907872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10916 00000100:00080000:2.0:1788574162.907877:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574162.907880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569804416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574162.907887:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574162.907904:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574162.907922:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569804544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574162.907935:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574162.907939:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574162.907941:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574162.907949:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10916 00000100:00080000:2.0:1788574162.907962:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574162.907967:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574162.907972:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574162.907973:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574162.908002:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569804672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574162.908011:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574162.908014:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574162.908016:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574162.908024:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10916 00000100:00080000:2.0:1788574162.908031:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574162.908037:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574162.908044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574162/real 1788574162] req@ffff920a60ac4700 x1875444569804416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574162.908045:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574162.908061:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569804800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574162.908069:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574162.908071:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569804416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574162.908078:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574162.908084:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574162.908088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574162.908093:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10916 00000100:00080000:0.0:1788574162.908096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574162.908102:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569804416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574162.908103:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574162.908114:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574162.908121:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574162.908131:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569804928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574162.908138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574162.908172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574162.908186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574162/real 1788574162] req@ffff920a60ac4a80 x1875444569804544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574162.908195:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569804544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574162.908204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574162.908207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574162.908209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574162.908218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574162.908237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574162.908246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574162/real 1788574162] req@ffff920a60ac5500 x1875444569804672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574162.908256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569804672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574162.908261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574162.908263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574162.908266:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574162.908270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574162.908285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574162.908293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574162/real 1788574162] req@ffff920a60ac6300 x1875444569804800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574162.908302:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569804800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574162.908309:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574162.908313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574162.908317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574162.908336:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574162.908349:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574162.908358:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574162/real 1788574162] req@ffff920a60ac7100 x1875444569804928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574162.908370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569804928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574162.908376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574162.908379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574162.908381:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569804928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574163.355755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574163.931720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574164.379750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574164.955673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574165.403755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574165.979716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574166.427772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574167.003690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574167.451756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574168.027699:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574168.027702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574168.027712:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574168.027717:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.027729:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10921 00000100:00080000:2.0:1788574168.027738:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.027747:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574168.027789:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569805056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.027813:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.027815:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574168.027817:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.027834:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10921 00000100:00080000:2.0:1788574168.027838:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.027842:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574168.027853:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574168.027864:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569805184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.027872:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.027875:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574168.027876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569805056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574168.027877:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.027884:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10921 00000100:00080000:2.0:1788574168.027888:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.027890:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574168.027893:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574168.027902:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569805312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.027908:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.027911:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574168.027912:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.027914:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10921 00000100:00080000:2.0:1788574168.027916:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.027919:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574168.027922:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574168.027928:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569805440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574168.027935:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574168.027937:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574168.027938:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574168.027940:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10921 00000100:00080000:2.0:1788574168.027943:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574168.027945:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574168.027948:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574168.027953:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569805568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574168.027966:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574168.028013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff920a60ac5180 x1875444569805056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574168.028043:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569805056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.028055:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574168.028058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574168.028062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569805056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574168.028089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574168.028111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574168.028122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff920a60ac6a00 x1875444569805184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574168.028134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569805184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.028142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574168.028145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574168.028148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574168.028153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574168.028169:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574168.028196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff920a60ac7100 x1875444569805312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574168.028205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569805312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.028218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574168.028221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574168.028223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574168.028228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574168.028244:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574168.028269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff920a60ac6300 x1875444569805440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574168.028278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569805440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.028285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574168.028287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574168.028290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574168.028295:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574168.028309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574168.028334:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574168/real 1788574168] req@ffff920a60ac5500 x1875444569805568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574168.028344:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569805568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574168.028350:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574168.028353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574168.028356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574168.475805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574169.051691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574169.499724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574170.075729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574170.523813:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574171.099707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574171.547776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574172.123698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574172.571796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574173.147694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574173.147705:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.147715:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574173.147720:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.147732:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10926 00000100:00080000:2.0:1788574173.147744:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.147752:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574173.147796:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569805696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.147822:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.147825:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574173.147826:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.147830:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10926 00000100:00080000:2.0:1788574173.147832:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.147834:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574173.147844:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574173.147854:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569805824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.147862:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.147866:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574173.147868:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574173.147870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569805696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574173.147882:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10926 00000100:00080000:2.0:1788574173.147885:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.147890:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574173.147896:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574173.147910:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569805952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.147920:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574173.147924:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574173.147926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.147931:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10926 00000100:00080000:2.0:1788574173.147935:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.147940:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574173.147946:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574173.147961:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569806080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574173.147972:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574173.147973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574173.147979:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574173.147982:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574173.147988:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10926 00000100:00080000:2.0:1788574173.147993:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574173.147998:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574173.148005:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574173.148018:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569806208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574173.148028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff920a60ac4a80 x1875444569805696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574173.148048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569805696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574173.148058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574173.148062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574173.148065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569805696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574173.148088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574173.148119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574173.148134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff920a60ac4700 x1875444569805824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574173.148149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569805824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574173.148158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574173.148162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574173.148166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574173.148173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574173.148193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574173.148202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff920a60ac4000 x1875444569805952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574173.148210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569805952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574173.148215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574173.148218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574173.148220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569805952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574173.148223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574173.148235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574173.148241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff920a60ac5c00 x1875444569806080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574173.148249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569806080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574173.148254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574173.148255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574173.148257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574173.148261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574173.148271:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574173.148277:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574173/real 1788574173] req@ffff920a60ac4380 x1875444569806208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574173.148290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569806208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574173.148294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574173.148296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574173.148298:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574173.595730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574174.171678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574174.619870:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574175.195663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574175.643791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574176.219691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574176.667785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574177.243707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574177.691718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574178.267705:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574178.267706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574178.267716:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574178.267721:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.267731:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10931 00000100:00080000:3.0:1788574178.267741:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.267749:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574178.267789:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569806336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574178.267828:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.267833:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574178.267835:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.267840:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10931 00000100:00080000:3.0:1788574178.267844:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.267850:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574178.267861:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574178.267876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569806336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574178.267877:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569806464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574178.267894:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574178.267897:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574178.267899:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.267904:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10931 00000100:00080000:3.0:1788574178.267908:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.267912:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574178.267919:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574178.267931:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569806592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574178.267942:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574178.267946:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574178.267949:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574178.267953:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574178.267959:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10931 00000100:00080000:3.0:1788574178.267966:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.267971:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574178.267978:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574178.267979:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff920a529b1c00 x1875444569806336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574178.267993:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569806720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574178.267998:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569806336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574178.268018:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574178.268020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574178.268025:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574178.268028:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574178.268030:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574178.268033:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569806336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574178.268039:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10931 00000100:00080000:3.0:1788574178.268045:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574178.268049:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574178.268055:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574178.268058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574178.268072:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569806848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574178.268081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574178.268097:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff920a529b3480 x1875444569806464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574178.268112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569806464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574178.268130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574178.268134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574178.268138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574178.268145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574178.268172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574178.268183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff920a529b2a00 x1875444569806592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574178.268192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569806592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574178.268197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574178.268200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574178.268202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574178.268206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574178.268220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574178.268226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff920a529b2d80 x1875444569806720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574178.268263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569806720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574178.268273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574178.268275:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574178.268277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574178.268290:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574178.268304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574178.268312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574178/real 1788574178] req@ffff920a529b3100 x1875444569806848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574178.268319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569806848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574178.268324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574178.268326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574178.268327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569806848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574178.715722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574179.291695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574179.739768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574180.315679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574180.763726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574181.339711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574181.787797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574182.363673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574182.811757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574183.387697:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574183.387704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574183.387709:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574183.387715:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574183.387726:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10936 00000100:00080000:3.0:1788574183.387737:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574183.387745:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574183.387793:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569806976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574183.387827:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574183.387832:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574183.387834:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574183.387838:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10936 00000100:00080000:3.0:1788574183.387841:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574183.387846:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574183.387856:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574183.387871:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569807104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574183.387873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569806976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574183.387887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574183.387890:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574183.387892:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574183.387895:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10936 00000100:00080000:3.0:1788574183.387897:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574183.387900:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574183.387903:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574183.387911:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569807232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574183.387922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574183.387925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574183.387927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574183.387929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10936 00000100:00080000:3.0:1788574183.387958:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574183.387962:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574183.387966:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574183.387975:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569807360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574183.387981:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574183.387988:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574183.387991:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574183.387992:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574183.387995:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10936 00000100:00080000:3.0:1788574183.387998:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574183.388000:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574183.388002:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574183.388010:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569807488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574183.388034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff920a529b1f80 x1875444569806976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.388057:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569806976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.388070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574183.388076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574183.388080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569806976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574183.388109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574183.388134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574183.388148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff920a529b0000 x1875444569807104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.388162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569807104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.388171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574183.388175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574183.388179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574183.388186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574183.388205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574183.388216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff920a529b0700 x1875444569807232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.388229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569807232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.388238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574183.388242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574183.388246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574183.388253:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574183.388274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574183.388284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff920a529b2300 x1875444569807360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.388298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569807360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.388308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574183.388311:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574183.388315:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574183.388322:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574183.388342:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574183.388353:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574183/real 1788574183] req@ffff920a529b0e00 x1875444569807488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574183.388366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569807488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574183.388375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574183.388379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574183.388382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574183.835819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574184.411724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574184.859739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574185.435714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574185.883809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574186.459679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574186.907762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574187.483665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574187.931824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574188.507667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574188.507706:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574188.507715:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574188.507720:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574188.507730:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10941 00000100:00080000:0.0:1788574188.507741:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574188.507750:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574188.507793:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569807616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574188.507820:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574188.507823:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574188.507824:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574188.507828:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10941 00000100:00080000:0.0:1788574188.507830:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574188.507833:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574188.507843:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574188.507852:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569807744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574188.507861:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574188.507864:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574188.507867:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574188.507873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569807616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574188.507878:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10941 00000100:00080000:0.0:1788574188.507889:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574188.507895:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574188.507900:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574188.507910:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569807872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574188.507922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574188.507925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574188.507926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574188.507929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10941 00000100:00080000:0.0:1788574188.507931:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574188.507934:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574188.507937:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574188.507946:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569808000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574188.507952:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574188.507954:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574188.507955:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574188.507957:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10941 00000100:00080000:0.0:1788574188.507960:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574188.507963:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574188.507966:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574188.507973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574188.507975:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569808128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574188.508021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff920a48350380 x1875444569807616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574188.508041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569807616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574188.508053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574188.508058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574188.508062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569807616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574188.508089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574188.508131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574188.508145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff920a48350700 x1875444569807744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574188.508158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569807744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574188.508167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574188.508171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574188.508175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574188.508182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574188.508203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574188.508215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff920a48353480 x1875444569807872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574188.508227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569807872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574188.508235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574188.508239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574188.508242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569807872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574188.508248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574188.508268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574188.508278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff920a48351500 x1875444569808000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574188.508291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569808000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574188.508299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574188.508302:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574188.508305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574188.508312:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574188.508332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574188.508344:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574188/real 1788574188] req@ffff920a48350a80 x1875444569808128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574188.508356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569808128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574188.508364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574188.508368:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574188.508371:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574188.955768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574189.531692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574189.979768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574190.555684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574191.003746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574191.579724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574192.027769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574192.603706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574193.051740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574193.627708:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574193.627708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574193.627721:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574193.627726:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574193.627737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10947 00000100:00080000:0.0:1788574193.627749:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574193.627758:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574193.627801:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569808256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574193.627834:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574193.627838:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574193.627841:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574193.627845:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10947 00000100:00080000:0.0:1788574193.627850:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574193.627854:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574193.627865:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574193.627880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569808384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574193.627894:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574193.627900:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574193.627901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569808256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574193.627902:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574193.627914:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10947 00000100:00080000:0.0:1788574193.627919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574193.627924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574193.627931:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574193.627946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569808512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574193.627957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574193.627962:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574193.627965:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574193.627970:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10947 00000100:00080000:0.0:1788574193.627974:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574193.627980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574193.627983:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574193.627990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574193.628007:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569808640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574193.628018:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574193.628023:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574193.628025:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574193.628030:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10947 00000100:00100000:2.0:1788574193.628036:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574193/real 1788574193] req@ffff920a48353b80 x1875444569808256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574193.628037:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574193.628047:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574193.628058:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574193.628070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569808256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574193.628076:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569808768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574193.628095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574193.628102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574193.628108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569808256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574193.628132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574193.628158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574193.628172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574193/real 1788574193] req@ffff920a48353800 x1875444569808384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574193.628185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569808384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574193.628195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574193.628198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574193.628202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574193.628209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574193.628229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574193.628240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574193/real 1788574193] req@ffff920a48351c00 x1875444569808512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574193.628253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569808512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574193.628268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574193.628272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574193.628275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574193.628282:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574193.628301:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574193.628312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574193/real 1788574193] req@ffff920a48350a80 x1875444569808640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574193.628325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569808640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574193.628333:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574193.628336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574193.628355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574193.628363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574193.628384:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574193.628396:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574193/real 1788574193] req@ffff920a48351500 x1875444569808768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574193.628407:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569808768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574193.628414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574193.628417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574193.628421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569808768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574194.075776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574194.651664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574195.099720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574195.675693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574196.123726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574196.699646:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574197.147804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574197.723685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574198.171749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574198.747638:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574198.747715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574198.747726:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574198.747731:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574198.747744:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10952 00000100:00080000:2.0:1788574198.747758:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574198.747767:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574198.747811:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569808896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574198.747837:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574198.747841:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574198.747843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574198.747849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10952 00000100:00080000:2.0:1788574198.747856:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574198.747860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574198.747874:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574198.747890:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569808896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574198.747891:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569809024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574198.747916:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574198.747920:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574198.747921:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574198.747926:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10952 00000100:00080000:2.0:1788574198.747930:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574198.747933:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574198.747937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574198.747948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569809152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574198.747957:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574198.747960:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574198.747961:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574198.747964:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10952 00000100:00080000:2.0:1788574198.747966:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574198.747972:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574198.747978:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574198.747979:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574198.747994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569809280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574198.748002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574198.748006:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574198.748007:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574198.748013:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10952 00000100:00080000:2.0:1788574198.748019:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574198.748023:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574198.748028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574198/real 1788574198] req@ffff920a60ac7800 x1875444569808896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574198.748031:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574198.748063:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569809408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574198.748072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569808896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574198.748126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574198.748132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574198.748138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569808896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574198.748167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574198.748191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574198.748210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574198/real 1788574198] req@ffff920a60ac4e00 x1875444569809024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574198.748221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569809024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574198.748228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574198.748230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574198.748235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574198.748243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574198.748270:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574198.748289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574198/real 1788574198] req@ffff920a60ac4380 x1875444569809152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574198.748305:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569809152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574198.748312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574198.748315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574198.748319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574198.748330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574198.748359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574198.748378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574198/real 1788574198] req@ffff920a60ac5c00 x1875444569809280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574198.748389:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569809280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574198.748395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574198.748397:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574198.748400:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574198.748404:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574198.748416:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574198.748422:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574198/real 1788574198] req@ffff920a60ac4000 x1875444569809408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574198.748429:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569809408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574198.748434:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574198.748436:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574198.748437:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574199.195756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574199.771676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574200.219750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574200.795714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574201.243862:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574201.819680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574202.267731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574202.843719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574203.291844:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574203.867702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574203.867708:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574203.867719:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574203.867725:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574203.867737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10957 00000100:00080000:2.0:1788574203.867760:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574203.867770:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574203.867817:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569809536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574203.867839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574203.867842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574203.867844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574203.867847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10957 00000100:00080000:2.0:1788574203.867851:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574203.867854:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574203.867864:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574203.867886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569809536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574203.867886:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569809664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574203.867913:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574203.867915:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574203.867917:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574203.867924:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10957 00000100:00080000:2.0:1788574203.867929:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574203.867934:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574203.867941:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574203.867959:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569809792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574203.867971:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574203.867975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574203.867978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788574203.867984:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574203.867989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10957 00000100:00080000:2.0:1788574203.867997:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574203.868003:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574203.868010:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574203.868025:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569809920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574203.868037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574203/real 1788574203] req@ffff920a60ac4700 x1875444569809536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574203.868053:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574203.868060:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1788574203.868066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569809536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574203.868066:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574203.868082:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10957 00000100:00080000:2.0:1788574203.868095:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574203.868097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574203.868103:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788574203.868111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:2.0:1788574203.868114:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574203.868122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569809536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574203.868130:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569810048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574203.868170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574203.868195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574203.868206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574203/real 1788574203] req@ffff920a60ac4a80 x1875444569809664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574203.868216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569809664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574203.868222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574203.868225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574203.868227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574203.868232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574203.868243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574203.868249:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574203/real 1788574203] req@ffff920a60ac5500 x1875444569809792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574203.868257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569809792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574203.868267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574203.868269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574203.868271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574203.868275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574203.868286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574203.868291:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574203/real 1788574203] req@ffff920a60ac6300 x1875444569809920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574203.868298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569809920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574203.868303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574203.868305:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574203.868307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569809920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574203.868311:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574203.868321:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574203.868326:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574203/real 1788574203] req@ffff920a60ac7100 x1875444569810048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574203.868333:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569810048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574203.868338:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574203.868340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574203.868342:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574204.315777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574204.891746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574205.339769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574205.915734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574206.363761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574206.939721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574207.387783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574207.963717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574208.411747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574208.987682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574208.987682:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574208.987697:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574208.987702:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574208.987715:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10962 00000100:00080000:3.0:1788574208.987728:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574208.987736:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574208.987799:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569810176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574208.987829:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574208.987833:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574208.987836:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574208.987842:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10962 00000100:00080000:3.0:1788574208.987846:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574208.987852:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574208.987864:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574208.987866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569810176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574208.987881:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569810304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574208.987894:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574208.987898:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574208.987900:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574208.987904:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10962 00000100:00080000:3.0:1788574208.987908:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574208.987912:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574208.987918:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574208.987931:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569810432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574208.987942:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574208.987945:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574208.987947:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574208.987952:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10962 00000100:00080000:3.0:1788574208.987956:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574208.987961:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574208.987968:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574208.987981:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569810560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574208.987991:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810304:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574208.987995:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574208.988001:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574208.988003:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574208.988009:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10962 00000100:00080000:3.0:1788574208.988015:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574208.988020:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574208.988026:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574208.988032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574208.988043:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569810688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574208.988068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574208.988088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574209.435779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574210.011761:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574210.459702:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574211.035741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574211.483790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574212.059888:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788574212.059909:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788574212.059953:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574212.059970:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574212.060009:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574212.060019:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574212.060029:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574212.060037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574208/real 1788574212] req@ffff920a529b3b80 x1875444569810304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788574212.060043:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574212.060070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569810304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574212.060083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574212.060088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574212.060093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574212.060101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574208/real 1788574212] req@ffff920a529b1500 x1875444569810432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574212.060115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569810432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574212.060123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574212.060127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574212.060131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574212.060139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574208/real 1788574212] req@ffff920a529b1180 x1875444569810176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574212.060153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569810176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574212.060161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574212.060164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574212.060168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569810176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574212.060192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574208/real 1788574212] req@ffff920a529b0380 x1875444569810560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574212.060205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569810560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574212.060213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574212.060217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574212.060220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574212.060227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574208/real 1788574212] req@ffff920a529b0e00 x1875444569810688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574212.060239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569810688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574233 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574212.060247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574212.060251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574212.060254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574212.507725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574213.083701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574213.531756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574214.107709:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574214.107720:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574214.107725:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574214.107735:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10967 00000100:00080000:1.0:1788574214.107747:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574214.107749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574214.107759:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574214.107818:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569810816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574214.107851:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574214.107855:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574214.107857:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574214.107862:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10967 00000100:00080000:1.0:1788574214.107867:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574214.107873:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574214.107893:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574214.107900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569810816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574214.107914:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569810944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574214.107928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574214.107932:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574214.107934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574214.107939:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10967 00000100:00080000:1.0:1788574214.107945:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574214.107950:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574214.107955:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574214.107973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569811072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574214.107984:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574214.107991:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574214.107992:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574214.107998:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574214.108004:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10967 00000100:00080000:1.0:1788574214.108009:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574214.108014:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574214.108022:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574214.108037:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569811200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574214.108045:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff920a52890700 x1875444569810816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574214.108052:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574214.108061:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574214.108065:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574214.108072:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10967 00000100:00080000:1.0:1788574214.108077:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574214.108082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569810816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574214.108086:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574214.108097:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574214.108103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.108112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574214.108116:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569811328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574214.108127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569810816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574214.108183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574214.108211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574214.108225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff920a52893100 x1875444569810944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.108238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569810944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.108246:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.108250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574214.108253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569810944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574214.108260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574214.108289:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574214.108301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff920a52891180 x1875444569811072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.108312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569811072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.108320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.108323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574214.108327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574214.108333:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574214.108356:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574214.108367:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff920a52892300 x1875444569811200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.108379:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569811200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.108388:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.108391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574214.108395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574214.108401:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574214.108422:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574214.108434:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574214/real 1788574214] req@ffff920a52892d80 x1875444569811328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574214.108447:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569811328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574239 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574214.108457:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574214.108460:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574214.108464:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574214.555727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574215.131761:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574215.579744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574216.155662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574216.603808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574217.179684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574217.627743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574218.203700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574218.651827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574219.227679:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574219.227695:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574219.227700:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574219.227704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574219.227723:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10972 00000100:00080000:1.0:1788574219.227737:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574219.227746:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574219.227795:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569811456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574219.227825:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574219.227830:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574219.227832:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574219.227837:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10972 00000100:00080000:1.0:1788574219.227842:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574219.227847:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574219.227873:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574219.227875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569811456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574219.227892:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569811584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574219.227901:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574219.227904:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574219.227905:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574219.227909:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10972 00000100:00080000:1.0:1788574219.227912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574219.227915:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574219.227920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574219.227927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569811712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574219.227934:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574219.227937:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574219.227938:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574219.227940:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10972 00000100:00080000:1.0:1788574219.227942:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574219.227945:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574219.227948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574219.227955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569811840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574219.227961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574219.227967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574219.227970:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574219.227972:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574219.227975:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10972 00000100:00080000:1.0:1788574219.227978:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574219.227981:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574219.227984:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574219.227992:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569811968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574219.228004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff920a52891880 x1875444569811456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.228020:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569811456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.228029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.228033:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574219.228036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569811456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574219.228059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574219.228076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574219.228085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff920a52890e00 x1875444569811584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.228099:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569811584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.228104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.228106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574219.228108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.228112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574219.228124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574219.228130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff920a52890380 x1875444569811712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.228138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569811712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.228143:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.228145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574219.228146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574219.228150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574219.228161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574219.228166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff920a52893480 x1875444569811840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.228174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569811840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.228183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.228186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574219.228188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574219.228191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574219.228201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574219.228207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574219/real 1788574219] req@ffff920a52893800 x1875444569811968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574219.228213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569811968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574244 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574219.228218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574219.228220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574219.228222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569811968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574219.675772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574220.251721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574220.699718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574221.275666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574221.723736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574222.299728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574222.747797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574223.323705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574223.771788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574224.347714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574224.347742:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574224.347750:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574224.347754:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574224.347762:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10977 00000100:00080000:3.0:1788574224.347771:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574224.347778:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574224.347815:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569812096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574224.347844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574224.347848:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574224.347849:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574224.347853:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10977 00000100:00080000:3.0:1788574224.347857:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574224.347861:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574224.347869:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574224.347876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569812096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574224.347884:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569812224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574224.347892:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574224.347895:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574224.347897:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574224.347900:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10977 00000100:00080000:3.0:1788574224.347902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574224.347904:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574224.347908:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574224.347916:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569812352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574224.347922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574224.347924:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574224.347926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574224.347929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10977 00000100:00080000:3.0:1788574224.347931:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574224.347933:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574224.347937:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574224.347942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574224.347946:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569812480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574224.347953:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574224.347957:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574224.347958:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574224.347961:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10977 00000100:00080000:3.0:1788574224.347965:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574224.347967:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574224.347970:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574224.347977:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569812608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574224.347983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff920a529b2300 x1875444569812096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.348005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569812096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.348016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.348020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574224.348025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569812096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574224.348047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574224.348077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574224.348090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff920a529b0700 x1875444569812224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.348102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569812224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.348111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.348114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574224.348118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574224.348124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574224.348139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574224.348147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff920a529b0000 x1875444569812352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.348159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569812352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.348175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.348178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574224.348182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574224.348188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574224.348202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574224.348210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff920a529b1f80 x1875444569812480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.348221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569812480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.348230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.348232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574224.348236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574224.348241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574224.348254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574224.348263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574224/real 1788574224] req@ffff920a529b3100 x1875444569812608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574224.348274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569812608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574249 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574224.348282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574224.348285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574224.348289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574224.795781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574225.371703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574225.819766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574226.395698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574226.843773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574227.419720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574227.867726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574228.443674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574228.891779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574229.467681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574229.467684:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574229.467693:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574229.467698:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574229.467709:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10982 00000100:00080000:2.0:1788574229.467719:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574229.467727:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574229.467769:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569812736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574229.467797:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574229.467799:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574229.467801:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574229.467804:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10982 00000100:00080000:2.0:1788574229.467806:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574229.467809:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574229.467818:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574229.467827:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569812864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574229.467835:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574229.467839:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574229.467841:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574229.467843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569812736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574229.467847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10982 00000100:00080000:2.0:1788574229.467854:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574229.467858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574229.467864:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574229.467879:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569812992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574229.467891:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574229.467895:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574229.467897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574229.467902:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10982 00000100:00080000:2.0:1788574229.467906:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574229.467909:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574229.467922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574229.467933:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569813120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574229.467939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574229.467945:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574229.467948:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574229.467949:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574229.467953:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10982 00000100:00080000:2.0:1788574229.467956:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574229.467958:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574229.467961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574229.467970:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569813248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574229.467995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff920a60ac6a00 x1875444569812736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574229.468018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569812736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574229.468030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574229.468034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574229.468040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569812736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574229.468067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574229.468095:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574229.468109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff920a60ac5180 x1875444569812864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574229.468122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569812864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574229.468131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574229.468135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574229.468144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574229.468151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574229.468170:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574229.468181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff920a60ac7100 x1875444569812992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574229.468193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569812992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574229.468201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574229.468204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574229.468207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569812992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574229.468213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574229.468230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574229.468241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff920a60ac6300 x1875444569813120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574229.468251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569813120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574229.468259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574229.468262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574229.468265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574229.468285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574229.468305:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574229.468316:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574229/real 1788574229] req@ffff920a60ac5500 x1875444569813248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574229.468327:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569813248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574254 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574229.468335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574229.468338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574229.468341:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574229.915767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574230.491709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574230.939779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574231.515663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574231.963725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574232.539726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574232.987761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574233.563658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574234.011730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574234.587669:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574234.587670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574234.587681:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574234.587686:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574234.587696:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10987 00000100:00080000:2.0:1788574234.587707:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574234.587716:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574234.587764:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569813376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574234.587799:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574234.587805:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574234.587807:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574234.587813:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10987 00000100:00080000:2.0:1788574234.587818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574234.587825:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574234.587837:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574234.587845:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569813376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574234.587854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569813504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574234.587864:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574234.587867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574234.587870:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574234.587876:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10987 00000100:00080000:2.0:1788574234.587881:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574234.587887:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574234.587894:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574234.587910:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569813632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574234.587919:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574234.587925:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574234.587928:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574234.587931:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574234.587935:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10987 00000100:00080000:2.0:1788574234.587940:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574234.587945:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574234.587951:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574234.587957:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff920a60ac4a80 x1875444569813376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574234.587965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569813760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574234.587978:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574234.587981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569813376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574234.587986:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574234.587991:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574234.587998:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574234.588000:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10987 00000100:00080000:3.0:1788574234.588005:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788574234.588008:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574234.588010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569813376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574234.588019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574234.588025:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574234.588063:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569813888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574234.588143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574234.588166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574234.588176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff920a60ac4700 x1875444569813504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.588190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569813504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.588196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.588199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574234.588201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574234.588205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574234.588217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574234.588223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff920a60ac4000 x1875444569813632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.588231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569813632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.588236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.588238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574234.588240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574234.588243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574234.588398:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574234.588410:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff920a60ac5c00 x1875444569813760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.588419:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569813760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.588465:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.588469:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574234.588472:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574234.588477:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574234.588493:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574234.588502:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574234/real 1788574234] req@ffff920a60ac4380 x1875444569813888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574234.588509:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569813888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574259 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574234.588514:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574234.588516:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574234.588518:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569813888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574235.035765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574235.611689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574236.059789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574236.635680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574237.083751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574237.659757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574238.107744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574238.683699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574239.131727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574239.707672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574239.707699:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574239.707706:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574239.707710:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574239.707721:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10993 00000100:00080000:1.0:1788574239.707737:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574239.707745:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574239.707788:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569814016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574239.707814:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574239.707817:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574239.707818:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574239.707822:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10993 00000100:00080000:1.0:1788574239.707824:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574239.707827:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574239.707837:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574239.707850:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569814144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574239.707862:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574239.707868:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788574239.707869:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569814016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574239.707873:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574239.707881:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10993 00000100:00080000:1.0:1788574239.707885:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574239.707890:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574239.707896:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574239.707907:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569814272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574239.707915:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574239.707918:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574239.707920:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574239.707922:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10993 00000100:00080000:1.0:1788574239.707926:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574239.707929:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574239.707933:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574239.707940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569814400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574239.707947:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574239.707950:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574239.707951:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574239.707966:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10993 00000100:00080000:1.0:1788574239.707971:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574239.707974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574239.707979:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574239.707986:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574239.708001:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569814528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574239.708031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574239/real 1788574239] req@ffff920a52892a00 x1875444569814016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574239.708052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569814016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574239.708066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574239.708070:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574239.708075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569814016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574239.708105:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574239.708131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574239.708144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574239/real 1788574239] req@ffff920a52891f80 x1875444569814144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574239.708157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569814144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574239.708164:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574239.708168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574239.708171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574239.708177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574239.708195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574239.708205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574239/real 1788574239] req@ffff920a52891500 x1875444569814272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574239.708217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569814272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574239.708225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574239.708229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574239.708233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574239.708239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574239.708256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574239.708264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574239/real 1788574239] req@ffff920a52892680 x1875444569814400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574239.708272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569814400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574239.708278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574239.708280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574239.708282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574239.708286:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574239.708296:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574239.708301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574239/real 1788574239] req@ffff920a52890a80 x1875444569814528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574239.708312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569814528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574264 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574239.708320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574239.708324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574239.708327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574240.155795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574240.731676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574241.179777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574241.755719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574242.203797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574242.779665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574243.227776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574243.803736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574244.251784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574244.827694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574244.827712:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574244.827724:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574244.827730:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574244.827744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 10998 00000100:00080000:1.0:1788574244.827756:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574244.827763:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574244.827809:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569814656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574244.827836:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574244.827839:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574244.827840:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574244.827850:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10998 00000100:00080000:1.0:1788574244.827859:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574244.827865:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574244.827891:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574244.827896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569814656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574244.827912:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569814784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574244.827929:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574244.827934:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574244.827936:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574244.827943:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10998 00000100:00080000:1.0:1788574244.827947:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574244.827951:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574244.827956:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574244.827964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574244.827971:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569814912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574244.827980:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574244.827983:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574244.827984:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574244.827990:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10998 00000100:00080000:1.0:1788574244.827996:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574244.828000:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574244.828005:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574244.828007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574244/real 1788574244] req@ffff920a52890000 x1875444569814656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574244.828016:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569815040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574244.828027:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574244.828029:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574244.828033:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574244.828035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569814656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574244.828040:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 10998 00000100:00080000:1.0:1788574244.828045:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574244.828052:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574244.828057:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574244.828070:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569815168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574244.828073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574244.828088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574244.828094:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569814656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574244.828119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574244.828141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574244.828152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574244/real 1788574244] req@ffff920a52893b80 x1875444569814784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574244.828160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569814784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574244.828166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574244.828168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574244.828171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574244.828175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574244.828186:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574244.828193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574244/real 1788574244] req@ffff920a52890a80 x1875444569814912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574244.828206:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569814912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574244.828214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574244.828218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574244.828222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569814912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574244.828229:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574244.828251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574244.828260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574244/real 1788574244] req@ffff920a52892680 x1875444569815040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574244.828268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569815040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574244.828273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574244.828276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574244.828278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574244.828281:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574244.828293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574244.828299:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574244/real 1788574244] req@ffff920a52891500 x1875444569815168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574244.828306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569815168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574269 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574244.828326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574244.828331:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574244.828333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574245.275766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574245.851715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574246.299812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574246.875716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574247.323712:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574247.899693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574248.347771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574248.923696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574249.371827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574249.947737:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574249.947740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574249.947754:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574249.947761:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574249.947775:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11003 00000100:00080000:2.0:1788574249.947790:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574249.947814:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574249.947866:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569815296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574249.947899:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574249.947903:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574249.947906:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574249.947913:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11003 00000100:00080000:2.0:1788574249.947918:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574249.947922:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574249.947942:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569815296:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574249.947943:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574249.947973:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569815424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574249.947998:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574249.948001:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574249.948003:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574249.948009:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11003 00000100:00080000:2.0:1788574249.948025:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574249.948030:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574249.948036:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574249.948039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574249.948053:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569815552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574249.948061:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574249.948064:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574249.948066:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574249.948070:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11003 00000100:00080000:2.0:1788574249.948075:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574249.948079:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574249.948084:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574249.948093:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569815680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574249.948099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574249/real 1788574249] req@ffff920a60ac4e00 x1875444569815296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574249.948110:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574249.948124:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574249.948127:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574249.948139:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11003 00000100:00080000:2.0:1788574249.948143:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574249.948147:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574249.948152:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574249.948157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569815296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574249.948161:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569815808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574249.948230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574249.948237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574249.948246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569815296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574249.948273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574249.948303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574249.948318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574249/real 1788574249] req@ffff920a60ac7800 x1875444569815424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574249.948328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569815424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574249.948335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574249.948338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574249.948340:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574249.948345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574249.948361:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574249.948367:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574249/real 1788574249] req@ffff920a60ac4380 x1875444569815552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574249.948374:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569815552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574249.948380:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574249.948381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574249.948383:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574249.948387:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574249.948398:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574249.948404:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574249/real 1788574249] req@ffff920a60ac5c00 x1875444569815680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574249.948412:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569815680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574249.948416:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574249.948418:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574249.948421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574249.948425:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574249.948435:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574249.948440:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574249/real 1788574249] req@ffff920a60ac4000 x1875444569815808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574249.948447:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569815808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574274 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574249.948452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574249.948454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574249.948456:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569815808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574250.395777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574250.971720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574251.419791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574251.995683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574252.443717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574253.019680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574253.467701:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574254.043665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574254.491738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574255.067659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574255.067661:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574255.067669:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574255.067674:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574255.067683:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11008 00000100:00080000:2.0:1788574255.067693:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574255.067700:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574255.067756:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569815936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574255.067781:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574255.067784:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574255.067786:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574255.067789:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11008 00000100:00080000:2.0:1788574255.067792:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574255.067794:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574255.067802:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574255.067812:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569816064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574255.067821:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574255.067823:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574255.067828:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574255.067828:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569815936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574255.067835:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11008 00000100:00080000:2.0:1788574255.067838:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574255.067840:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574255.067843:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574255.067852:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569816192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574255.067858:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574255.067861:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574255.067862:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574255.067865:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11008 00000100:00080000:2.0:1788574255.067893:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574255.067895:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574255.067898:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574255.067905:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574255.067908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569816320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574255.067916:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574255.067919:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574255.067920:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574255.067923:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11008 00000100:00080000:2.0:1788574255.067928:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574255.067930:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574255.067934:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574255.067941:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569816448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574255.067944:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff920a60ac4700 x1875444569815936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.067971:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569815936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.067980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.067983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574255.067987:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569815936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574255.068005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574255.068023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574255.068031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff920a60ac4a80 x1875444569816064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.068039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569816064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.068045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.068047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574255.068049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574255.068053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574255.068064:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574255.068069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff920a60ac5500 x1875444569816192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.068076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569816192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.068082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.068084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574255.068085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574255.068089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574255.068100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574255.068106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff920a60ac6300 x1875444569816320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.068113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569816320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.068118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.068120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574255.068122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574255.068125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574255.068134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574255.068142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574255/real 1788574255] req@ffff920a60ac7100 x1875444569816448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574255.068154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569816448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574280 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574255.068161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574255.068174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574255.068178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574255.515792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574256.091672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574256.539733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574257.115739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574257.563738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574258.139721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574258.587795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574259.163725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574259.611820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574260.187759:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574260.187773:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574260.187779:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574260.187791:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11013 00000100:00100000:0.0:1788574260.187792:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574260.187804:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574260.187813:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574260.187849:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569816576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574260.187871:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574260.187873:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574260.187875:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574260.187878:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11013 00000100:00080000:3.0:1788574260.187881:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574260.187883:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574260.187891:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574260.187901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569816704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574260.187909:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574260.187912:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574260.187916:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569816576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574260.187917:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574260.187925:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11013 00000100:00080000:3.0:1788574260.187928:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574260.187930:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574260.187933:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574260.187942:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569816832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574260.187953:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574260.187956:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574260.187957:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574260.187960:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11013 00000100:00080000:3.0:1788574260.187963:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574260.187965:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574260.187969:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574260.187976:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569816960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574260.187982:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574260.187985:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574260.187986:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574260.187989:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574260.187993:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11013 00000100:00080000:3.0:1788574260.187995:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574260.187998:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574260.188002:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574260.188011:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569817088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574260.188020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574260] req@ffff920a529b2d80 x1875444569816576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574260.188035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569816576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574260.188045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574260.188048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574260.188051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569816576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574260.188069:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574260.188084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574260.188092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574260] req@ffff920a529b2a00 x1875444569816704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574260.188103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569816704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574260.188112:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574260.188115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574260.188119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574260.188125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574260.188146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574260.188158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574260] req@ffff920a529b3480 x1875444569816832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574260.188170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569816832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574260.188179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574260.188182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574260.188185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574260.188192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574260.188214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574260.188226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574260] req@ffff920a529b1c00 x1875444569816960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574260.188239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569816960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574260.188247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574260.188251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574260.188254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569816960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574260.188260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574260.188281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574260.188292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574260/real 1788574260] req@ffff920a529b3100 x1875444569817088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574260.188304:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569817088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574285 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574260.188312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574260.188315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574260.188319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574260.635785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574261.211751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574261.659777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574262.235742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574262.683778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574263.259718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574263.707740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574264.283700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574264.731734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574265.307715:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574265.307724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574265.307732:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574265.307739:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.307752:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11018 00000100:00080000:3.0:1788574265.307766:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.307776:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574265.307822:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569817216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.307850:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.307854:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574265.307856:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.307860:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11018 00000100:00080000:3.0:1788574265.307864:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.307869:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574265.307880:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574265.307894:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569817344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.307904:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.307908:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574265.307910:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569817216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574265.307910:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.307914:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11018 00000100:00080000:3.0:1788574265.307918:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.307921:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574265.307926:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574265.307935:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569817472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.307943:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.307947:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574265.307949:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.307953:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11018 00000100:00080000:3.0:1788574265.307956:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.307960:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574265.307965:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574265.307974:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569817600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574265.307982:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574265.307985:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574265.307987:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574265.307991:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11018 00000800:00000100:0.0:1788574265.307993:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574265.307995:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574265.307998:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574265.308003:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574265.308011:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569817728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574265.308032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff920a529b1f80 x1875444569817216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574265.308054:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569817216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574265.308066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574265.308071:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574265.308076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569817216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574265.308102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574265.308122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574265.308132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff920a529b0000 x1875444569817344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574265.308140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569817344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574265.308146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574265.308148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574265.308150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574265.308154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574265.308165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574265.308171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff920a529b0700 x1875444569817472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574265.308178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569817472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574265.308188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574265.308191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574265.308193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574265.308196:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574265.308207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574265.308212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff920a529b2300 x1875444569817600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574265.308219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569817600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574265.308225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574265.308227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574265.308229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574265.308232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574265.308242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574265.308247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574265/real 1788574265] req@ffff920a529b0e00 x1875444569817728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574265.308254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569817728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574290 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574265.308259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574265.308261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574265.308263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574265.755803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574266.331703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574266.779766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574267.355691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574267.803791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574268.379698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574268.827778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574269.403698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574269.851788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574270.427708:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574270.427717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574270.427719:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574270.427729:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574270.427743:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11023 00000100:00080000:1.0:1788574270.427756:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574270.427764:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574270.427804:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569817856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574270.427836:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574270.427842:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574270.427844:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574270.427853:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11023 00000100:00080000:1.0:1788574270.427858:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574270.427864:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574270.427877:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574270.427886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569817856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574270.427897:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569817984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574270.427914:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574270.427918:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574270.427921:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574270.427926:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11023 00000100:00080000:1.0:1788574270.427932:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574270.427938:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574270.427944:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574270.427956:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569818112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574270.427967:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574270.427970:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574270.427972:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574270.427975:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11023 00000100:00080000:1.0:1788574270.427978:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574270.427982:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574270.427988:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574270.427999:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569818240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574270.428009:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574270.428012:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574270.428014:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574270.428019:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11023 00000100:00080000:1.0:1788574270.428023:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574270.428027:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574270.428033:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574270.428033:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574270.428051:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569818368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574270.428079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff920a52891f80 x1875444569817856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.428239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569817856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.428258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.428263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574270.428268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569817856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574270.428348:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574270.428379:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574270.428395:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff920a52892a00 x1875444569817984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.428408:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569817984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.428417:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.428420:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574270.428424:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569817984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574270.428431:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574270.428453:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574270.428466:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff920a52893800 x1875444569818112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.428493:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569818112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.428503:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.428507:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574270.428510:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574270.428518:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574270.428541:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574270.428554:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff920a52893480 x1875444569818240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.428567:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569818240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.428576:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.428580:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574270.428584:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574270.428625:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574270.428650:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574270.428664:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574270/real 1788574270] req@ffff920a52890380 x1875444569818368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574270.428677:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569818368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574295 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574270.428686:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574270.428698:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574270.428705:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574270.875747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574271.451719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574271.899761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574272.475706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574272.923717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574273.499717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574273.947749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574274.523713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574274.971798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574275.547707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574275.547714:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574275.547723:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574275.547729:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574275.547744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11028 00000100:00080000:3.0:1788574275.547757:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574275.547768:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574275.547831:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569818496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574275.547864:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574275.547868:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574275.547870:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574275.547875:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11028 00000100:00080000:3.0:1788574275.547879:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574275.547885:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574275.547899:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569818496:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574275.547906:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574275.547927:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569818624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574275.547939:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574275.547943:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574275.547945:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574275.547950:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11028 00000100:00080000:3.0:1788574275.547954:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574275.547959:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574275.547965:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574275.547979:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569818752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574275.547997:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574275.548001:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574275.548003:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574275.548009:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11028 00000100:00080000:3.0:1788574275.548014:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574275.548019:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574275.548026:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574275.548027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574275.548044:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569818880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574275.548059:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574275.548063:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574275.548066:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574275.548071:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11028 00000100:00100000:0.0:1788574275.548074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818752:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574275.548077:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574275.548085:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574275.548093:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574275.548108:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569819008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574275.548124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574275.548157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574275.995726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574276.571716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574277.019782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574277.595738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574278.043721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574278.619700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788574278.619843:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788574278.619878:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574278.619890:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574278.619920:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574278.619928:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574278.619935:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574278.619942:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788574278.619963:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574278] req@ffff920a529b0e00 x1875444569819008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574278.619991:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569819008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574278.620004:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574278.620008:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574278.620011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819008:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574278.620018:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574278] req@ffff920a529b3b80 x1875444569818880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574278.620031:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569818880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574278.620037:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574278.620039:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574278.620041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574278.620045:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574278] req@ffff920a529b1180 x1875444569818624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574278.620053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569818624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574278.620058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574278.620061:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574278.620065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574278.620078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574278] req@ffff920a529b1500 x1875444569818752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574278.620093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569818752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574278.620099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574278.620102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574278.620105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569818752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574278.620111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574275/real 1788574278] req@ffff920a529b0380 x1875444569818496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574278.620124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569818496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574300 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574278.620131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574278.620135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574278.620138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569818496:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788574279.067775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574279.643693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574280.091789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574280.667725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574280.667744:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574280.667756:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574280.667760:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574280.667772:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11034 00000100:00080000:3.0:1788574280.667785:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574280.667794:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574280.667842:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569819136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574280.667874:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574280.667878:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574280.667885:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574280.667890:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11034 00000100:00080000:3.0:1788574280.667894:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574280.667899:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574280.667910:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574280.667927:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569819264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574280.667940:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574280.667944:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574280.667947:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574280.667952:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569819136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574280.667956:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11034 00000100:00080000:3.0:1788574280.667964:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574280.667970:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574280.667975:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574280.667990:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569819392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574280.668000:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574280.668005:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574280.668008:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574280.668012:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11034 00000100:00080000:3.0:1788574280.668017:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574280.668022:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574280.668029:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574280.668037:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574280.668045:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569819520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574280.668057:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574280.668061:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574280.668063:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574280.668069:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11034 00000100:00080000:3.0:1788574280.668074:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574280.668078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574280/real 1788574280] req@ffff920a529b2300 x1875444569819136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574280.668083:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574280.668091:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574280.668108:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569819648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574280.668111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569819136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574280.668136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574280.668141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574280.668145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569819136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574280.668173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574280.668199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574280.668215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574280/real 1788574280] req@ffff920a529b0700 x1875444569819264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574280.668227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569819264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574280.668236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574280.668239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574280.668242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574280.668263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574280.668286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574280.668299:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574280/real 1788574280] req@ffff920a529b0000 x1875444569819392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574280.668311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569819392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574280.668360:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574280.668364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574280.668367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574280.668373:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574280.668403:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574280.668416:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574280/real 1788574280] req@ffff920a529b1f80 x1875444569819520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574280.668427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569819520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574280.668434:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574280.668437:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574280.668440:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574280.668453:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574280.668473:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574280.668483:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574280/real 1788574280] req@ffff920a529b3100 x1875444569819648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574280.668495:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569819648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574305 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574280.668503:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574280.668506:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574280.668509:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574281.115791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574281.691669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574282.139835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574282.715674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574283.163826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574283.739721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574284.187756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574284.763742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574285.211778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574285.787679:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574285.787689:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574285.787695:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574285.787707:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11039 00000100:00100000:2.0:1788574285.787712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574285.787720:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574285.787730:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574285.787768:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569819776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574285.787794:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574285.787798:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574285.787800:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574285.787804:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11039 00000100:00080000:3.0:1788574285.787809:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574285.787815:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574285.787827:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574285.787835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569819776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574285.787845:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569819904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574285.787859:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574285.787864:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574285.787867:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574285.787871:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11039 00000100:00080000:3.0:1788574285.787876:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574285.787881:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574285.787888:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574285.787901:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569820032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574285.787906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574285.787914:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574285.787917:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574285.787919:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574285.787924:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11039 00000100:00080000:3.0:1788574285.787928:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574285.787931:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574285.787935:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574285.787943:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569820160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574285.787950:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574285.787953:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574285/real 1788574285] req@ffff920a529b1c00 x1875444569819776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574285.787954:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574285.787957:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574285.787965:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11039 00000100:00080000:3.0:1788574285.787969:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574285.787975:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574285.787979:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574285.787985:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569819776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574285.787990:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569820288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574285.788006:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574285.788013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574285.788018:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569819776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574285.788046:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574285.788072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574285.788083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574285/real 1788574285] req@ffff920a529b3480 x1875444569819904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574285.788098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569819904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574285.788105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574285.788107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574285.788110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569819904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574285.788114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574285.788130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574285.788141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574285/real 1788574285] req@ffff920a529b2a00 x1875444569820032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574285.788154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569820032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574285.788173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574285.788177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574285.788180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574285.788185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574285.788203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574285.788211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574285/real 1788574285] req@ffff920a529b2d80 x1875444569820160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574285.788219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569820160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574285.788228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574285.788231:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574285.788233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574285.788251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574285.788266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574285.788273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574285/real 1788574285] req@ffff920a529b3100 x1875444569820288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574285.788281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569820288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574310 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574285.788286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574285.788288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574285.788290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574286.235768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574286.811682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574287.259756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574287.835691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574288.283780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574288.859707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574289.307809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574289.883685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574290.331706:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574290.907687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574290.907722:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574290.907730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574290.907734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574290.907744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11044 00000100:00080000:0.0:1788574290.907755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574290.907764:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574290.907810:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569820416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574290.907841:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574290.907846:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574290.907849:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574290.907855:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11044 00000100:00080000:0.0:1788574290.907860:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574290.907865:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574290.907872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569820416:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574290.907878:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574290.907908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569820544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574290.907922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574290.907926:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574290.907928:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574290.907931:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574290.907938:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11044 00000100:00080000:0.0:1788574290.907944:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574290.907950:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574290.907956:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574290.907964:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574290/real 1788574290] req@ffff920a48353480 x1875444569820416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574290.907974:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569820672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574290.907982:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569820416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574290.907995:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574290.907997:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574290.908003:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574290.908004:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788574290.908009:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574290.908010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569820416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574290.908019:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11044 00000100:00080000:0.0:1788574290.908025:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574290.908030:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574290.908035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820544:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788574290.908040:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574290.908054:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574290.908065:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574290/real 1788574290] req@ffff920a48350700 x1875444569820544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574290.908072:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569820800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574290.908078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569820544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574290.908097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574290.908102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788574290.908104:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574290.908106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574290.908112:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788574290.908116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820672:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574290.908117:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574290.908125:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11044 00000100:00080000:0.0:1788574290.908130:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574290.908134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574290.908139:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574290.908145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574290/real 1788574290] req@ffff920a48350380 x1875444569820672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574290.908147:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574290.908179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569820672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574290.908184:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569820928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574290.908201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574290.908209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574290.908213:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574290.908227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574290.908245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574290.908254:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574290/real 1788574290] req@ffff920a48353480 x1875444569820800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574290.908262:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569820800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574290.908268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574290.908270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574290.908272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574290.908275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574290.908285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574290.908291:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574290/real 1788574290] req@ffff920a48351500 x1875444569820928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574290.908298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569820928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574315 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574290.908303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574290.908305:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574290.908307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569820928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574291.355775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574291.931676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574292.379716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574292.955702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574293.403745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574293.979675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574294.427800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574295.003701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574295.451791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574296.027741:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574296.027750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574296.027751:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574296.027757:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574296.027767:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11049 00000100:00080000:1.0:1788574296.027777:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574296.027784:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574296.027826:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569821056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574296.027846:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574296.027849:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574296.027851:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574296.027854:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11049 00000100:00080000:1.0:1788574296.027857:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574296.027859:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574296.027868:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574296.027876:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569821184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574296.027885:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574296.027888:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574296.027889:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574296.027895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569821056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574296.027896:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11049 00000100:00080000:1.0:1788574296.027901:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574296.027906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574296.027909:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574296.027919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569821312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574296.027926:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574296.027929:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574296.027930:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574296.027932:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11049 00000100:00080000:1.0:1788574296.027935:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574296.027938:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574296.027941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574296.027948:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569821440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574296.027954:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574296.027956:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574296.027959:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788574296.027960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574296.027964:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11049 00000100:00080000:1.0:1788574296.027968:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574296.027970:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574296.027973:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574296.027981:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569821568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574296.028005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff920a52890e00 x1875444569821056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574296.028019:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569821056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574296.028028:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574296.028031:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574296.028034:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569821056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574296.028052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574296.028069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574296.028092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff920a52891880 x1875444569821184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574296.028102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569821184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574296.028108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574296.028110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574296.028113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574296.028117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574296.028132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574296.028140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff920a52892d80 x1875444569821312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574296.028147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569821312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574296.028153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574296.028154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574296.028156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574296.028160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574296.028172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574296.028178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff920a52892300 x1875444569821440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574296.028186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569821440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574296.028191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574296.028193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574296.028194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574296.028198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574296.028209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574296.028219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574296/real 1788574296] req@ffff920a52891180 x1875444569821568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574296.028231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569821568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574321 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574296.028239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574296.028242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574296.028246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574296.475771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574297.051727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574297.499770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574298.075748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574298.523834:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574299.099674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574299.547765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574300.123706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574300.571737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574301.147699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574301.147705:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.147714:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574301.147718:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.147733:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11054 00000100:00080000:3.0:1788574301.147746:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.147753:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574301.147795:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569821696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.147821:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.147824:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574301.147825:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.147828:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11054 00000100:00080000:3.0:1788574301.147830:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.147833:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574301.147843:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574301.147852:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569821824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.147859:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.147861:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574301.147862:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.147865:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11054 00000100:00080000:3.0:1788574301.147867:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.147871:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574301.147876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569821696:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574301.147876:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574301.147884:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569821952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.147896:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.147899:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574301.147900:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.147903:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11054 00000100:00080000:3.0:1788574301.147905:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.147908:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574301.147911:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574301.147919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569822080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574301.147925:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574301.147928:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574301.147930:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574301.147932:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11054 00000100:00080000:3.0:1788574301.147934:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574301.147936:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574301.147939:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574301.147945:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569822208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574301.147968:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574301.148007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff920a529b1f80 x1875444569821696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574301.148029:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569821696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574301.148042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574301.148046:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574301.148051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569821696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574301.148060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574301.148081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574301.148119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff920a529b0000 x1875444569821824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574301.148128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569821824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574301.148135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574301.148137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574301.148140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574301.148163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574301.148184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574301.148192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff920a529b0700 x1875444569821952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574301.148199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569821952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574301.148205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574301.148207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574301.148209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569821952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574301.148213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574301.148224:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574301.148230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff920a529b2300 x1875444569822080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574301.148238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569822080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574301.148243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574301.148245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574301.148247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574301.148250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574301.148260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574301.148266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574301/real 1788574301] req@ffff920a529b0380 x1875444569822208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574301.148275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569822208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574326 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574301.148283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574301.148286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574301.148290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574301.595757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574302.171714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574302.619814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574303.195667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574303.643800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574304.219735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574304.667764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574305.243737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574305.691717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574306.267751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574306.268396:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.268404:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574306.268409:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.268419:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11059 00000100:00080000:3.0:1788574306.268431:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.268440:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574306.268483:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569822336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574306.268514:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.268518:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574306.268520:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.268526:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11059 00000100:00080000:3.0:1788574306.268530:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.268534:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574306.268546:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574306.268564:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569822464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574306.268572:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569822336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574306.268580:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574306.268671:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574306.268718:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff920a529b1500 x1875444569822336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574306.268738:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569822336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574306.268751:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574306.268756:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574306.268761:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569822336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574306.268768:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574306.268773:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.268780:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11059 00000100:00080000:3.0:1788574306.268786:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574306.268790:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822464:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574306.268796:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574306.268804:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574306.268822:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574306.268824:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569822592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574306.268838:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff920a529b1180 x1875444569822464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574306.268840:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.268850:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574306.268854:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.268862:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11059 00000100:00100000:0.0:1788574306.268865:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569822464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574306.268868:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574306.268879:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788574306.268881:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574306.268888:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000008:00100000:3.0:1788574306.268888:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574306.268895:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574306.268909:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569822720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574306.268912:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822592:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574306.268931:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574306.268936:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788574306.268940:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574306.268941:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574306.268949:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11059 00000100:00080000:3.0:1788574306.268955:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574306.268958:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff920a529b3b80 x1875444569822592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574306.268963:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574306.268974:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574306.268981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569822592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574306.268991:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569822848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574306.268994:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574306.269002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574306.269010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574306.269025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574306.269048:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574306.269059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff920a529b0e00 x1875444569822720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574306.269071:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569822720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574306.269079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574306.269083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574306.269087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574306.269096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574306.269121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574306.269148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574306/real 1788574306] req@ffff920a529b1180 x1875444569822848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574306.269163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569822848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574331 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574306.269173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574306.269176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574306.269180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569822848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574306.715806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574307.291680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574307.739763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574308.315715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574308.763758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574309.339685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574309.787744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574310.363700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574310.811789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574311.387644:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574311.387654:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574311.387658:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574311.387668:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11064 00000100:00080000:3.0:1788574311.387677:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574311.387684:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574311.387724:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569822976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574311.387737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574311.387754:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574311.387760:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574311.387762:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574311.387769:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11064 00000100:00080000:3.0:1788574311.387774:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574311.387780:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574311.387792:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574311.387810:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569823104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574311.387813:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569822976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574311.387830:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574311.387834:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574311.387835:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574311.387839:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11064 00000100:00080000:3.0:1788574311.387843:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574311.387845:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574311.387850:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574311.387868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569823232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574311.387876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574311.387878:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574311.387880:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574311.387882:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11064 00000100:00080000:3.0:1788574311.387885:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574311.387890:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574311.387891:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574311.387898:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574311.387906:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569823360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574311.387913:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574311.387916:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574311.387917:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574311.387920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11064 00000100:00080000:3.0:1788574311.387922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574311.387925:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574311.387928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574311.387936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff920a529b1500 x1875444569822976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574311.387937:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569823488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574311.387975:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569822976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574311.387987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574311.387992:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574311.387996:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569822976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574311.388021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574311.388048:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574311.388062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff920a529b0380 x1875444569823104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574311.388076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569823104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574311.388085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574311.388089:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574311.388093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574311.388099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574311.388119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574311.388131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff920a529b2300 x1875444569823232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574311.388151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569823232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574311.388159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574311.388163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574311.388167:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574311.388173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574311.388229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574311.388242:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff920a529b0700 x1875444569823360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574311.388256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569823360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574311.388264:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574311.388268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574311.388271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574311.388277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574311.388299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574311.388312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574311/real 1788574311] req@ffff920a529b0000 x1875444569823488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574311.388324:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569823488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574336 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574311.388339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574311.388342:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574311.388346:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574311.835721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574312.411696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574312.859776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574313.435743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574313.883806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574314.459679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574314.907718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574315.483659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574315.931798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574316.507671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574316.507677:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574316.507684:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574316.507689:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574316.507700:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11069 00000100:00080000:2.0:1788574316.507710:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574316.507719:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574316.507762:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569823616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574316.507783:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574316.507786:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574316.507787:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574316.507791:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11069 00000100:00080000:2.0:1788574316.507794:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574316.507797:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574316.507805:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574316.507814:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569823744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574316.507822:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569823616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574316.507824:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574316.507831:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574316.507833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574316.507837:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11069 00000100:00080000:2.0:1788574316.507839:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574316.507841:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574316.507852:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574316.507861:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569823872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574316.507868:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574316.507870:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574316.507872:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574316.507875:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11069 00000100:00080000:2.0:1788574316.507877:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574316.507879:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574316.507883:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574316.507883:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574316.507894:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569824000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574316.507901:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574316.507903:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574316.507905:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574316.507908:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11069 00000100:00080000:2.0:1788574316.507911:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574316.507913:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574316.507917:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574316.507918:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff920a60ac5180 x1875444569823616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574316.507927:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569824128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574316.507936:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569823616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574316.507945:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574316.507949:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574316.507952:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569823616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574316.507969:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574316.507984:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574316.507992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff920a60ac6a00 x1875444569823744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574316.508000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569823744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574316.508005:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574316.508007:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574316.508010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574316.508013:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574316.508023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574316.508029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff920a60ac7100 x1875444569823872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574316.508036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569823872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574316.508041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574316.508043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574316.508044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569823872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574316.508047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574316.508057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574316.508063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff920a60ac6300 x1875444569824000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574316.508073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569824000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574316.508081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574316.508084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574316.508088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574316.508094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574316.508115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574316.508126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574316/real 1788574316] req@ffff920a60ac5500 x1875444569824128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574316.508134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569824128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574341 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574316.508139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574316.508141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574316.508143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574316.955755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574317.531746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574317.979781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574318.555674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574319.003757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574319.579681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574320.027764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574320.603672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574321.051793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574321.627708:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574321.627708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574321.627720:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574321.627725:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574321.627734:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11075 00000100:00080000:2.0:1788574321.627744:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574321.627751:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574321.627788:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569824256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574321.627815:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574321.627819:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574321.627821:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574321.627826:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11075 00000100:00080000:2.0:1788574321.627830:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574321.627834:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574321.627843:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574321.627856:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569824384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574321.627867:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574321.627871:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574321.627873:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574321.627878:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11075 00000100:00080000:2.0:1788574321.627882:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574321.627887:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574321.627893:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574321.627915:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569824256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574321.627921:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569824512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574321.627935:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574321.627939:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574321.627941:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574321.627946:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11075 00000100:00080000:2.0:1788574321.627950:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574321.627955:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574321.627960:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574321.627974:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569824640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574321.627976:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574321.627989:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574321.627993:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574321.627995:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574321.628008:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11075 00000100:00080000:2.0:1788574321.628013:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574321.628017:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574321.628021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574321/real 1788574321] req@ffff920a60ac4a80 x1875444569824256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574321.628024:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574321.628040:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569824768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574321.628049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569824256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574321.628064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574321.628069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574321.628074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569824256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574321.628085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574321.628111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574321.628137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574321/real 1788574321] req@ffff920a60ac4700 x1875444569824384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574321.628151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569824384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574321.628158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574321.628160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574321.628163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574321.628181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574321.628196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574321.628206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574321/real 1788574321] req@ffff920a60ac4000 x1875444569824512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574321.628219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569824512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574321.628228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574321.628232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574321.628235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574321.628242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574321.628264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574321.628276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574321/real 1788574321] req@ffff920a60ac5c00 x1875444569824640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574321.628323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569824640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574321.628333:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574321.628337:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574321.628341:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574321.628347:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574321.628370:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574321.628383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574321/real 1788574321] req@ffff920a60ac4380 x1875444569824768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574321.628396:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569824768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574346 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574321.628405:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574321.628409:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574321.628413:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569824768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574322.075743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574322.651681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574323.099757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574323.675742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574324.123796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574324.699726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574325.147774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574325.723686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574326.171750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574326.747686:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574326.747687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574326.747699:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574326.747705:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574326.747716:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11080 00000100:00080000:2.0:1788574326.747727:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574326.747736:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574326.747776:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569824896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574326.747810:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574326.747814:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574326.747817:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574326.747821:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11080 00000100:00080000:2.0:1788574326.747826:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574326.747830:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574326.747841:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574326.747857:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569825024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574326.747867:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569824896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574326.747875:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574326.747880:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574326.747883:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574326.747888:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11080 00000100:00080000:2.0:1788574326.747893:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574326.747899:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574326.747905:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574326.747920:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569825152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574326.747932:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574326.747937:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574326.747939:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574326.747943:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574326.747949:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11080 00000100:00080000:2.0:1788574326.747954:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574326.747960:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574326.747967:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574326.747977:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574326/real 1788574326] req@ffff920a60ac7800 x1875444569824896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574326.747983:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569825280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574326.748003:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574326.748007:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569824896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574326.748009:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574326.748014:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574326.748024:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11080 00000100:00080000:0.0:1788574326.748029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574326.748030:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574326.748038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788574326.748040:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574326.748046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569824896:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574326.748049:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574326.748065:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569825408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574326.748072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574326.748103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574326.748116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574326/real 1788574326] req@ffff920a60ac4e00 x1875444569825024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574326.748129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569825024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574326.748138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574326.748142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574326.748146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574326.748151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574326.748167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574326.748176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574326/real 1788574326] req@ffff920a60ac4380 x1875444569825152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574326.748187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569825152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574326.748196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574326.748200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574326.748203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574326.748209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574326.748225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574326.748234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574326/real 1788574326] req@ffff920a60ac5c00 x1875444569825280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574326.748245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569825280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574326.748254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574326.748257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574326.748260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574326.748285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574326.748302:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574326.748311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574326/real 1788574326] req@ffff920a60ac4000 x1875444569825408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574326.748323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569825408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574326.748359:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574326.748363:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574326.748367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574327.195815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574327.771723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574328.219780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574328.795676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574329.243745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574329.819698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574330.267767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574330.843652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574331.291721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574331.867736:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574331.867736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574331.867748:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574331.867753:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574331.867763:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11085 00000100:00080000:1.0:1788574331.867774:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574331.867781:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574331.867822:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569825536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574331.867848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574331.867852:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574331.867853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574331.867857:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11085 00000100:00080000:1.0:1788574331.867860:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574331.867863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574331.867874:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574331.867880:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569825536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574331.867887:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569825664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574331.867895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574331.867898:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574331.867899:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574331.867902:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11085 00000100:00080000:1.0:1788574331.867904:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574331.867907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574331.867910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574331.867921:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569825792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574331.867927:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574331.867930:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574331.867932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574331.867934:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11085 00000100:00080000:1.0:1788574331.867947:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574331.867949:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574331.867954:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574331.867962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569825920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574331.867970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574331.867973:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574331.867977:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574331.867978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574331.867982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11085 00000100:00080000:1.0:1788574331.867984:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574331.867987:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574331.867990:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574331.867997:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569826048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574331.868025:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574331/real 1788574331] req@ffff920a52893100 x1875444569825536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574331.868048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569825536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574331.868061:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574331.868066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574331.868071:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569825536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574331.868100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574331.868133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574331.868148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574331/real 1788574331] req@ffff920a52890700 x1875444569825664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574331.868163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569825664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574331.868173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574331.868177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574331.868181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574331.868188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574331.868205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574331.868214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574331/real 1788574331] req@ffff920a52891180 x1875444569825792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574331.868227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569825792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574331.868243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574331.868246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574331.868250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574331.868257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574331.868277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574331.868285:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574331/real 1788574331] req@ffff920a52892300 x1875444569825920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574331.868293:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569825920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574331.868299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574331.868301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574331.868303:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569825920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574331.868307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574331.868327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574331.868337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574331/real 1788574331] req@ffff920a52892d80 x1875444569826048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574331.868349:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569826048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574331.868357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574331.868361:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574331.868365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574332.315718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574332.891672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574333.339781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574333.915683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574334.363801:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574334.939726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574335.387785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574335.963657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574336.411753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574336.987717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574336.987745:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574336.987752:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574336.987757:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574336.987768:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11090 00000100:00080000:0.0:1788574336.987779:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574336.987787:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574336.987825:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569826176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574336.987856:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574336.987861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574336.987863:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574336.987867:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11090 00000100:00080000:0.0:1788574336.987871:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574336.987875:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574336.987886:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574336.987901:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569826304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574336.987911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569826176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574336.987916:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574336.987922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574336.987925:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574336.987930:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11090 00000100:00080000:0.0:1788574336.987935:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574336.987940:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574336.987946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574336.987960:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569826432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574336.987970:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574336.987974:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574336.987976:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574336.987981:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11090 00000100:00080000:0.0:1788574336.987984:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574336.987988:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574336.987994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574336.988000:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574336.988009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569826560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574336.988021:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574336.988025:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574336.988027:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574336.988033:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11090 00000100:00080000:0.0:1788574336.988037:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574336.988044:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574336.988045:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574336/real 1788574336] req@ffff920a48350a80 x1875444569826176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574336.988051:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574336.988072:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569826688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574336.988076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569826176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574336.988102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574336.988110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574336.988114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569826176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574336.988134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574336.988156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574336.988166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574336/real 1788574336] req@ffff920a48351c00 x1875444569826304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574336.988174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569826304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574336.988180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574336.988183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574336.988185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574336.988189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574336.988201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574336.988207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574336/real 1788574336] req@ffff920a48353800 x1875444569826432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574336.988214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569826432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574336.988219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574336.988221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574336.988223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574336.988227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574336.988237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574336.988243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574336/real 1788574336] req@ffff920a48353b80 x1875444569826560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574336.988249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569826560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574336.988254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574336.988256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574336.988258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574336.988272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574336.988285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574336.988292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574336/real 1788574336] req@ffff920a48351500 x1875444569826688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574336.988299:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569826688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574361 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574336.988304:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574336.988306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574336.988307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574337.435788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574338.011672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574338.459773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574339.035757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574339.483758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574340.059700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574340.507793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574341.083724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574341.531753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574342.107755:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574342.107765:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574342.107770:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574342.107779:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11095 00000100:00080000:1.0:1788574342.107789:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574342.107792:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574342.107801:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574342.107840:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569826816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574342.107879:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574342.107884:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574342.107887:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574342.107893:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11095 00000100:00080000:1.0:1788574342.107898:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574342.107903:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574342.107913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574342.107915:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569826816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574342.107929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569826944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574342.107939:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574342.107943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574342.107946:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574342.107950:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11095 00000100:00080000:1.0:1788574342.107957:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574342.107963:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574342.107969:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574342.107986:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569827072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574342.107995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574342.107998:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574342.108000:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574342.108003:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11095 00000100:00080000:1.0:1788574342.108009:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574342.108013:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574342.108018:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574342.108027:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569827200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574342.108035:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574342.108038:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574342.108040:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574342.108043:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826944:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788574342.108048:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11095 00000100:00080000:1.0:1788574342.108054:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574342.108059:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574342.108065:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574342.108080:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569827328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574342.108088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574342.108119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574342.108147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574342.555713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574343.131816:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574343.579721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574344.155738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574344.603729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574345.179878:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574345.179996:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574345.180039:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574345.180055:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574345.180096:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574345.180107:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574345.180115:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574345.180124:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574345.180202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574345] req@ffff920a52893800 x1875444569827328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574345.180243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569827328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574345.180257:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574345.180263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574345.180268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827328:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574345.180278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574345] req@ffff920a52890e00 x1875444569826944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574345.180292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569826944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574345.180301:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574345.180306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574345.180310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569826944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574345.180319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574345] req@ffff920a52890380 x1875444569827072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574345.180338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569827072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574345.180347:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574345.180351:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574345.180355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574345.180363:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574345] req@ffff920a52893480 x1875444569827200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574345.180376:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569827200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574345.180384:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574345.180388:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574345.180391:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574345.180400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574342/real 1788574345] req@ffff920a52891880 x1875444569826816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574345.180429:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569826816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574345.180439:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574345.180443:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574345.180446:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569826816:192.168.202.157@tcp:250:kworker.0 00010000:00010000:0.0:1788574345.627736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574346.203726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574346.651714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574347.227715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574347.227723:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574347.227727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574347.227737:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11100 00000100:00080000:1.0:1788574347.227748:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574347.227756:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574347.227793:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569827456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574347.227809:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574347.227822:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574347.227826:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574347.227828:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574347.227834:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11100 00000100:00080000:1.0:1788574347.227839:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574347.227846:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574347.227857:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574347.227873:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569827456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574347.227875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569827584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574347.227893:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574347.227898:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574347.227900:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574347.227905:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11100 00000100:00080000:1.0:1788574347.227909:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574347.227913:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574347.227918:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574347.227931:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569827712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574347.227942:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574347.227948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574347.227950:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574347.227953:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574347.227960:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11100 00000100:00080000:1.0:1788574347.227965:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574347.227970:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574347.227976:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574347.227982:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff920a52892a00 x1875444569827456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574347.227992:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569827840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574347.228005:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574347.228008:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574347.228012:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569827456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574347.228013:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574347.228019:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11100 00000100:00080000:1.0:1788574347.228023:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574347.228027:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574347.228035:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574347.228044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788574347.228052:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569827968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574347.228053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574347.228064:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569827456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574347.228099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574347.228129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574347.228143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff920a52891f80 x1875444569827584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574347.228157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569827584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574347.228166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574347.228170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574347.228174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574347.228181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574347.228215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574347.228228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff920a52891500 x1875444569827712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574347.228241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569827712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574347.228251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574347.228254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574347.228258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574347.228264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574347.228288:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574347.228300:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff920a52892680 x1875444569827840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574347.228311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569827840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574347.228318:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574347.228322:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574347.228325:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574347.228331:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574347.228359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574347.228372:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574347/real 1788574347] req@ffff920a52890a80 x1875444569827968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574347.228383:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569827968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574372 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574347.228390:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574347.228393:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574347.228396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569827968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574347.675815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574348.251808:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574348.699711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574349.275690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574349.723764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574350.299676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574350.747717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574351.323690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574351.771749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574352.347706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574352.347715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574352.347722:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574352.347727:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574352.347738:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11105 00000100:00080000:0.0:1788574352.347749:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574352.347757:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574352.347801:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569828096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574352.347825:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574352.347828:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574352.347830:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574352.347833:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11105 00000100:00080000:0.0:1788574352.347835:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574352.347837:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574352.347855:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574352.347868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569828224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574352.347882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574352.347884:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569828096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574352.347886:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574352.347891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574352.347896:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11105 00000100:00080000:0.0:1788574352.347898:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574352.347901:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574352.347904:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574352.347913:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569828352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574352.347921:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574352.347924:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574352.347925:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574352.347928:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11105 00000100:00080000:0.0:1788574352.347930:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574352.347932:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574352.347936:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574352.347944:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569828480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574352.347950:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574352.347952:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574352.347953:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574352.347956:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11105 00000100:00080000:0.0:1788574352.347959:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574352.347962:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574352.347966:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574352.347970:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574352.347978:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569828608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574352.348002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff920a48353480 x1875444569828096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574352.348020:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569828096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574352.348031:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574352.348035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574352.348039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569828096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574352.348069:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574352.348087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574352.348096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff920a48350380 x1875444569828224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574352.348104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569828224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574352.348110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574352.348112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574352.348114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574352.348118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574352.348128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574352.348134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff920a48350700 x1875444569828352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574352.348142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569828352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574352.348147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574352.348149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574352.348150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574352.348154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574352.348166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574352.348172:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff920a48351500 x1875444569828480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574352.348186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569828480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574352.348191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574352.348194:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574352.348196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574352.348199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574352.348211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574352.348222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574352/real 1788574352] req@ffff920a48353b80 x1875444569828608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574352.348230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569828608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574377 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574352.348236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574352.348238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574352.348240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574352.795825:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574353.371671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574353.819782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574354.395784:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574354.843840:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574355.419702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574355.867764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574356.443737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574356.891859:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574357.467715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574357.467720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574357.467729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574357.467735:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574357.467746:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11110 00000100:00080000:0.0:1788574357.467757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574357.467765:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574357.467810:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569828736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574357.467837:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574357.467841:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574357.467842:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574357.467846:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11110 00000100:00080000:0.0:1788574357.467848:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574357.467851:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574357.467861:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574357.467870:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569828864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574357.467878:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569828736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574357.467894:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574357.467897:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574357.467898:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574357.467901:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11110 00000100:00080000:0.0:1788574357.467903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574357.467905:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574357.467908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574357.467917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569828992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574357.467923:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574357.467926:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574357.467927:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574357.467929:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11110 00000100:00080000:0.0:1788574357.467931:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574357.467934:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574357.467937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574357.467946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569829120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574357.467951:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574357.467953:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574357.467955:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574357.467957:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11110 00000100:00080000:0.0:1788574357.467960:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574357.467963:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574357.467967:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574357.467969:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574357.467977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569829248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574357.468011:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff920a48353800 x1875444569828736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574357.468028:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569828736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574357.468038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574357.468041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574357.468044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569828736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574357.468067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574357.468084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574357.468093:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff920a48351c00 x1875444569828864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574357.468102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569828864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574357.468108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574357.468111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574357.468115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574357.468122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574357.468142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574357.468154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff920a48350a80 x1875444569828992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574357.468167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569828992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574357.468180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574357.468182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574357.468184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569828992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574357.468189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574357.468205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574357.468212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff920a48353b80 x1875444569829120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574357.468219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569829120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574357.468225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574357.468226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574357.468228:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574357.468232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574357.468243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574357.468249:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574357/real 1788574357] req@ffff920a48351500 x1875444569829248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574357.468256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569829248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574382 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574357.468261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574357.468262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574357.468265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574357.915811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574358.491725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574358.939831:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574359.515716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574359.963805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574360.539737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574360.987821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574361.563686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574362.011817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574362.587714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574362.587720:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574362.587729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574362.587734:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574362.587745:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11115 00000100:00080000:0.0:1788574362.587757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574362.587765:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574362.587812:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569829376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574362.587839:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574362.587842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574362.587844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574362.587847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11115 00000100:00080000:0.0:1788574362.587850:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574362.587853:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574362.587865:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574362.587879:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569829504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574362.587881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569829376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574362.587903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574362.587905:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574362.587907:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574362.587910:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11115 00000100:00080000:0.0:1788574362.587912:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574362.587915:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574362.587918:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574362.587927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569829632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574362.587933:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574362.587936:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574362.587937:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574362.587941:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11115 00000100:00080000:0.0:1788574362.587945:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574362.587948:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574362.587951:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574362.587958:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569829760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574362.587963:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574362.587966:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574362.587970:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574362.587971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574362.587982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11115 00000100:00080000:0.0:1788574362.587986:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574362.587989:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574362.587993:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574362.588005:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569829888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574362.588035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff920a48350700 x1875444569829376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574362.588053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569829376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574362.588062:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574362.588066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574362.588069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569829376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574362.588090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574362.588112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574362.588120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff920a48350380 x1875444569829504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574362.588131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569829504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574362.588141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574362.588144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574362.588149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574362.588156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574362.588178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574362.588189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff920a48353480 x1875444569829632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574362.588197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569829632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574362.588203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574362.588205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574362.588207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574362.588212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574362.588227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574362.588238:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff920a48351500 x1875444569829760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574362.588249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569829760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574362.588258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574362.588262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574362.588266:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574362.588272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574362.588293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574362.588301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574362/real 1788574362] req@ffff920a48353b80 x1875444569829888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574362.588309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569829888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574387 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574362.588315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574362.588317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574362.588319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569829888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574363.035802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574363.611716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574364.059750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574364.635707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574365.083776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574365.659689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574366.107798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574366.683678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574367.131796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574367.707649:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574367.707693:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574367.707699:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574367.707704:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574367.707713:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11121 00000100:00080000:1.0:1788574367.707744:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574367.707751:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574367.707784:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569830016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574367.707805:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574367.707808:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574367.707809:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574367.707813:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11121 00000100:00080000:1.0:1788574367.707815:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574367.707818:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574367.707826:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574367.707835:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569830144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574367.707843:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574367.707848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574367.707850:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574367.707850:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569830016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574367.707856:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11121 00000100:00080000:1.0:1788574367.707861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574367.707864:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574367.707867:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574367.707881:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569830272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574367.707893:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574367.707896:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574367.707897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574367.707900:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11121 00000100:00080000:1.0:1788574367.707902:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574367.707904:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574367.707908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574367.707916:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569830400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574367.707924:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574367.707927:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574367.707931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574367.707932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574367.707935:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11121 00000100:00080000:1.0:1788574367.707939:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574367.707941:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574367.707944:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574367.707950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569830528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574367.707968:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574367/real 1788574367] req@ffff920a52893b80 x1875444569830016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574367.707990:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569830016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574367.708002:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574367.708007:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574367.708012:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569830016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574367.708034:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574367.708061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574367.708074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574367/real 1788574367] req@ffff920a52890000 x1875444569830144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574367.708087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569830144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574367.708096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574367.708100:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574367.708104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574367.708111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574367.708131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574367.708143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574367/real 1788574367] req@ffff920a52890a80 x1875444569830272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574367.708156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569830272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574367.708165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574367.708168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574367.708172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574367.708178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574367.708197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574367.708207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574367/real 1788574367] req@ffff920a52892680 x1875444569830400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574367.708219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569830400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574367.708228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574367.708231:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574367.708235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574367.708252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574367.708278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574367.708290:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574367/real 1788574367] req@ffff920a52891500 x1875444569830528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574367.708303:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569830528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574367.708312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574367.708315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574367.708318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574368.155760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574368.731722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574369.179746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574369.755655:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574370.203848:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574370.779738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574371.227733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574371.803656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574372.251788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574372.827689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574372.827720:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574372.827728:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574372.827733:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574372.827743:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11126 00000100:00080000:0.0:1788574372.827753:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574372.827761:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574372.827830:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569830656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574372.827858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574372.827862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574372.827865:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574372.827870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11126 00000100:00080000:0.0:1788574372.827874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574372.827879:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574372.827889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574372.827900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569830656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574372.827907:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569830784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574372.827920:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574372.827925:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574372.827927:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574372.827932:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11126 00000100:00080000:0.0:1788574372.827936:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574372.827941:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574372.827946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574372.827959:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569830912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574372.827969:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574372.827974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574372.827976:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574372.827980:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574372.827985:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11126 00000100:00080000:0.0:1788574372.827990:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574372.827995:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574372.828001:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574372.828016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574372/real 1788574372] req@ffff920a48350a80 x1875444569830656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574372.828018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569831040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574372.828050:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574372.828054:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574372.828057:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574372.828059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569830656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574372.828064:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11126 00000100:00080000:0.0:1788574372.828070:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574372.828078:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788574372.828080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788574372.828083:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574372.828089:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574372.828095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569830656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574372.828101:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569831168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574372.828124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574372.828152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574372.828165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574372/real 1788574372] req@ffff920a48351c00 x1875444569830784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574372.828186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569830784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574372.828195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574372.828198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574372.828201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574372.828208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574372.828229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574372.828240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574372/real 1788574372] req@ffff920a48353800 x1875444569830912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574372.828253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569830912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574372.828261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574372.828264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574372.828267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569830912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574372.828273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574372.828294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574372.828306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574372/real 1788574372] req@ffff920a48353b80 x1875444569831040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574372.828319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569831040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574372.828335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574372.828339:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574372.828343:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574372.828349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574372.828368:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574372.828379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574372/real 1788574372] req@ffff920a48351500 x1875444569831168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574372.828392:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569831168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574372.828402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574372.828406:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574372.828410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574373.275721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574373.851751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574374.299848:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574374.875693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574375.323824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574375.899720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574376.347794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574376.923675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574377.371799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574377.947719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574377.947733:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574377.947740:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574377.947744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574377.947756:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11131 00000100:00080000:0.0:1788574377.947768:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574377.947777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574377.947839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569831296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574377.947869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574377.947874:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574377.947876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574377.947881:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11131 00000100:00080000:0.0:1788574377.947885:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574377.947890:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574377.947902:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574377.947928:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569831424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574377.947928:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569831296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574377.947945:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574377.947948:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574377.947949:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574377.947952:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11131 00000100:00080000:0.0:1788574377.947955:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574377.947959:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574377.947963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574377.947972:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569831552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574377.947980:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574377.947982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574377.947983:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574377.947986:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11131 00000100:00080000:0.0:1788574377.947988:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574377.947991:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574377.947994:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574377.948001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569831680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574377.948008:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574377.948010:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574377.948011:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574377.948014:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11131 00000100:00080000:0.0:1788574377.948017:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574377.948022:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574377.948026:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574377.948034:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574377.948048:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569831808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574377.948071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574377/real 1788574377] req@ffff920a48353480 x1875444569831296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574377.948094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569831296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574377.948108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574377.948113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574377.948117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569831296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574377.948162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574377.948194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574377.948211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574377/real 1788574377] req@ffff920a48350380 x1875444569831424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574377.948226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569831424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574377.948235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574377.948239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574377.948242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574377.948249:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574377.948272:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574377.948285:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574377/real 1788574377] req@ffff920a48350700 x1875444569831552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574377.948298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569831552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574377.948308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574377.948313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574377.948318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574377.948325:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574377.948346:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574377.948359:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574377/real 1788574377] req@ffff920a48351500 x1875444569831680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574377.948371:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569831680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574377.948381:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574377.948385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574377.948389:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574377.948396:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574377.948427:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574377.948445:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574377/real 1788574377] req@ffff920a48353b80 x1875444569831808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574377.948463:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569831808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574402 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574377.948472:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574377.948478:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574377.948483:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569831808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574378.395860:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574378.971715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574379.419778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574379.995728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574380.443790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574381.019695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574381.467755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574382.043712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574382.491784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574383.067708:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574383.067708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574383.067725:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574383.067731:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.067744:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11136 00000100:00080000:0.0:1788574383.067757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.067767:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574383.067816:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569831936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574383.067845:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.067850:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574383.067852:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.067857:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11136 00000100:00080000:0.0:1788574383.067861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.067866:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574383.067892:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574383.067900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569831936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574383.067910:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569832064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574383.067924:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.067929:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574383.067931:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.067937:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11136 00000100:00080000:0.0:1788574383.067941:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.067946:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574383.067952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574383.067964:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569832192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574383.067976:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.067980:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574383.067982:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788574383.067989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574383.067990:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11136 00000100:00080000:0.0:1788574383.067996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.068001:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574383.068007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574383.068022:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569832320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574383.068038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff920a48353800 x1875444569831936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574383.068045:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574383.068054:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574383.068058:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574383.068065:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11136 00000100:00080000:0.0:1788574383.068070:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574383.068076:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574383.068076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569831936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574383.068084:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574383.068100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574383.068102:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569832448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574383.068108:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574383.068119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569831936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574383.068154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574383.068182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574383.068196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff920a48351c00 x1875444569832064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574383.068211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569832064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574383.068221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574383.068225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574383.068229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574383.068236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574383.068258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574383.068270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff920a48350a80 x1875444569832192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574383.068284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569832192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574383.068293:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574383.068297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574383.068301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574383.068307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574383.068330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574383.068343:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff920a48353b80 x1875444569832320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574383.068356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569832320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574383.068364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574383.068368:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574383.068372:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574383.068378:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574383.068400:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574383.068412:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574383/real 1788574383] req@ffff920a48351500 x1875444569832448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574383.068425:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569832448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574408 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574383.068443:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574383.068446:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574383.068450:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574383.515770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574384.091720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574384.539791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574385.115719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574385.563680:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574386.139696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574386.587770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574387.163678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574387.611707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574388.187685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574388.187698:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574388.187707:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574388.187711:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574388.187722:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11141 00000100:00080000:1.0:1788574388.187732:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574388.187739:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574388.187771:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569832576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574388.187794:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574388.187798:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574388.187799:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574388.187803:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11141 00000100:00080000:1.0:1788574388.187806:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574388.187809:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574388.187819:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574388.187830:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569832576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574388.187831:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569832704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574388.187844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574388.187847:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574388.187849:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574388.187854:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11141 00000100:00080000:1.0:1788574388.187857:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574388.187860:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574388.187871:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574388.187881:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569832832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574388.187889:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574388.187891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574388.187893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574388.187896:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11141 00000100:00080000:1.0:1788574388.187902:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574388.187902:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574388.187909:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574388.187913:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574388.187921:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569832960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574388.187927:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574388.187948:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574388.187949:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574388.187952:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11141 00000100:00080000:1.0:1788574388.187955:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574388.187959:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574388.187964:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574388.187974:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff920a52891f80 x1875444569832576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574388.187979:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569833088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574388.188011:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569832576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.188020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.188024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574388.188026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569832576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574388.188048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574388.188068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574388.188077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff920a52892a00 x1875444569832704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574388.188085:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569832704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.188091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.188093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574388.188095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574388.188099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574388.188119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574388.188127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff920a52891880 x1875444569832832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574388.188143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569832832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.188151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.188153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574388.188155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574388.188159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574388.188172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574388.188178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff920a52893480 x1875444569832960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574388.188185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569832960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.188190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.188192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574388.188194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569832960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574388.188198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574388.188213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574388.188224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574388/real 1788574388] req@ffff920a52890380 x1875444569833088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574388.188236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569833088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574413 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574388.188245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574388.188248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574388.188252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574388.635730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574389.211747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574389.659774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574390.235682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574390.683764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574391.259735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574391.707760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574392.283730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574392.731724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574393.307692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574393.307700:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574393.307705:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574393.307710:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574393.307719:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11146 00000100:00080000:2.0:1788574393.307729:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574393.307736:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574393.307770:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569833216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574393.307802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574393.307806:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574393.307808:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574393.307811:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11146 00000100:00080000:2.0:1788574393.307815:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574393.307818:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574393.307828:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574393.307836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569833216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574393.307839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569833344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574393.307852:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574393.307854:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574393.307855:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574393.307857:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11146 00000100:00080000:2.0:1788574393.307859:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574393.307861:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574393.307865:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574393.307872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569833472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574393.307878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574393.307880:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574393.307881:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574393.307884:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11146 00000100:00080000:2.0:1788574393.307898:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574393.307901:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574393.307905:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574393.307910:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574393.307918:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569833600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574393.307926:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574393.307929:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574393.307930:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574393.307933:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11146 00000100:00080000:2.0:1788574393.307935:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574393.307938:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574393.307941:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574393.307948:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574393] req@ffff920a60ac4700 x1875444569833216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574393.307950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569833728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574393.308004:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569833216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574393.308014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574393.308019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574393.308022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569833216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574393.308041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574393.308060:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574393.308073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574393] req@ffff920a60ac4a80 x1875444569833344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574393.308086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569833344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574393.308094:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574393.308098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574393.308101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574393.308107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574393.308127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574393.308137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574393] req@ffff920a60ac5500 x1875444569833472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574393.308149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569833472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574393.308163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574393.308166:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574393.308169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574393.308176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574393.308197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574393.308207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574393] req@ffff920a60ac6300 x1875444569833600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574393.308218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569833600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574393.308227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574393.308229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574393.308233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574393.308239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574393.308258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574393.308269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574393/real 1788574393] req@ffff920a60ac7100 x1875444569833728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574393.308282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569833728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574418 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574393.308290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574393.308293:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574393.308296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574393.755757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574394.331708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574394.779708:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574395.355751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574395.803742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574396.379733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574396.827741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574397.403692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574397.851750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574398.427702:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574398.427715:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788574398.427723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574398.427724:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574398.427741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11151 00000100:00080000:2.0:1788574398.427752:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574398.427758:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574398.427798:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569833856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574398.427830:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574398.427835:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574398.427837:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574398.427847:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11151 00000100:00080000:2.0:1788574398.427853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574398.427860:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574398.427869:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569833856:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574398.427872:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574398.427889:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569833984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574398.427898:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574398.427901:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574398.427903:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574398.427906:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11151 00000100:00080000:2.0:1788574398.427919:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574398.427921:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574398.427925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574398.427928:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574398.427946:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569834112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574398.427955:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574398.427957:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574398.427959:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574398.427964:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11151 00000100:00100000:3.0:1788574398.427965:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff920a60ac6a00 x1875444569833856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574398.427967:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574398.427974:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574398.427982:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574398.427991:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569833856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574398.427994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569834240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574398.428013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574398.428014:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574398.428020:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574398.428022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788574398.428023:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574398.428030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569833856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574398.428031:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11151 00000100:00080000:2.0:1788574398.428049:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574398.428055:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574398.428060:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574398.428069:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569834368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574398.428082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574398.428104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574398.428115:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff920a60ac5180 x1875444569833984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.428123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569833984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.428129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.428132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574398.428134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569833984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574398.428138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574398.428149:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574398.428155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff920a60ac7100 x1875444569834112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.428162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569834112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.428168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.428170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574398.428171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574398.428175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574398.428185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574398.428191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff920a60ac6300 x1875444569834240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.428199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569834240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.428203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.428205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574398.428207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574398.428216:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574398.428226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574398.428231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574398/real 1788574398] req@ffff920a60ac5500 x1875444569834368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574398.428241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569834368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574423 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574398.428251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574398.428255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574398.428258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574398.875742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574399.451693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574399.899830:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574400.475730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574400.923809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574401.499737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574401.947794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574402.523751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574402.971763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574403.547711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574403.547716:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.547725:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574403.547730:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.547744:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11156 00000100:00080000:1.0:1788574403.547757:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.547766:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574403.547813:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569834496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.547844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.547849:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574403.547851:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.547856:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11156 00000100:00080000:1.0:1788574403.547860:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.547865:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574403.547877:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574403.547889:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569834496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574403.547893:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569834624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.547906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.547910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574403.547912:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.547917:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11156 00000100:00080000:1.0:1788574403.547920:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.547924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574403.547930:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574403.547942:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569834752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.547952:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574403.547956:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574403.547958:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.547962:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11156 00000800:00000100:3.0:1788574403.547966:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574403.547969:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574403.547975:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574403.547981:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574403.547995:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569834880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574403.548005:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574403.548008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff920a52890e00 x1875444569834496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574403.548012:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574403.548016:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574403.548026:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11156 00000100:00080000:1.0:1788574403.548031:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574403.548034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569834496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574403.548047:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788574403.548047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.548054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:1.0:1788574403.548057:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574403.548058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569834496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574403.548075:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569835008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574403.548104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574403.548126:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574403.548137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff920a52893800 x1875444569834624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.548146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569834624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.548152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.548155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574403.548157:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574403.548162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574403.548174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574403.548183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff920a52892d80 x1875444569834752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.548196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569834752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.548205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.548209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574403.548213:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574403.548220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574403.548243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574403.548253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff920a52892300 x1875444569834880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.548261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569834880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.548267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.548269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574403.548271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569834880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574403.548283:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574403.548296:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574403.548303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574403/real 1788574403] req@ffff920a52891180 x1875444569835008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574403.548311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569835008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574428 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574403.548316:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574403.548317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574403.548319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574403.995826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574404.571717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574405.019781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574405.595724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574406.043750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574406.619740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574407.067759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574407.643742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574408.091794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574408.667765:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574408.667768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574408.667779:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574408.667784:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574408.667796:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11162 00000100:00080000:3.0:1788574408.667818:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574408.667827:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574408.667869:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569835136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574408.667901:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574408.667905:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574408.667907:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574408.667912:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11162 00000100:00080000:3.0:1788574408.667918:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574408.667924:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574408.667935:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574408.667948:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569835136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574408.667956:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569835264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574408.667991:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574408.667996:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574408.667998:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574408.668003:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11162 00000100:00080000:3.0:1788574408.668008:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574408.668013:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574408.668019:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574408.668052:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569835392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574408.668065:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574408.668069:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574408.668072:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574408.668089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835264:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574408.668090:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11162 00000100:00080000:3.0:1788574408.668100:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574408.668106:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574408.668113:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574408.668133:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569835520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574408.668159:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574408.668163:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574408.668166:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574408.668170:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11162 00000100:00080000:3.0:1788574408.668175:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574408.668180:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574408.668186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835392:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788574408.668197:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574408.668218:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569835648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574408.668235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574408.668277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574409.115818:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574409.691712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574410.139785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574410.715733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574411.163813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574411.739686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788574411.739961:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788574411.740009:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574411.740027:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574411.740141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574408/real 1788574411] req@ffff920a529b1f80 x1875444569835136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574433 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574411.740211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569835136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574433 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574411.740227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000800:00000100:0.0:1788574411.740237:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00080000:2.0:1788574411.740241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574411.740248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569835136:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788574411.740257:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574411.740286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574408/real 1788574411] req@ffff920a529b3100 x1875444569835264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1788574411.740287:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574411.740302:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574411.740312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569835264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574411.740322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574411.740326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574411.740330:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574411.740337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574408/real 1788574411] req@ffff920a529b2d80 x1875444569835392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574411.740348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569835392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574411.740356:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574411.740359:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574411.740362:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574411.740375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574408/real 1788574411] req@ffff920a529b3480 x1875444569835648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574411.740388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569835648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574411.740395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574411.740399:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574411.740402:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835648:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574411.740409:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574408/real 1788574411] req@ffff920a529b2a00 x1875444569835520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574411.740420:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569835520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574433 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574411.740428:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574411.740432:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574411.740435:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835520:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574412.187778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574412.763714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574413.211814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574413.787704:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574413.787715:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574413.787719:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574413.787730:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11167 00000100:00080000:3.0:1788574413.787742:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574413.787750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574413.787751:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574413.787798:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569835776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574413.787834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574413.787839:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574413.787841:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574413.787846:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11167 00000100:00080000:3.0:1788574413.787851:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574413.787856:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574413.787868:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574413.787876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569835776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574413.787886:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569835904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574413.787897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574413.787902:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574413.787905:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574413.787909:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11167 00000100:00080000:3.0:1788574413.787914:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574413.787918:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574413.787924:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574413.787939:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569836032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574413.787951:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574413.787955:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574413.787958:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574413.787963:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11167 00000800:00000100:0.0:1788574413.787970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574413.787970:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574413.787978:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574413.787985:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574413.787996:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569836160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574413.788005:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574413.788008:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574413.788010:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574413.788014:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11167 00000100:00080000:3.0:1788574413.788018:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574413.788029:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574413.788033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574413/real 1788574413] req@ffff920a529b1c00 x1875444569835776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574413.788039:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574413.788056:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569836288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574413.788065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569835776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574413.788081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574413.788088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574413.788093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569835776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574413.788134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574413.788162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574413.788176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574413/real 1788574413] req@ffff920a529b2a00 x1875444569835904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574413.788234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569835904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574413.788245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574413.788249:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574413.788253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569835904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574413.788260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574413.788289:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574413.788302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574413/real 1788574413] req@ffff920a529b3480 x1875444569836032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574413.788316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569836032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574413.788326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574413.788330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574413.788334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574413.788342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574413.788364:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574413.788375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574413/real 1788574413] req@ffff920a529b2d80 x1875444569836160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574413.788388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569836160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574413.788405:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574413.788409:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574413.788412:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574413.788419:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574413.788438:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574413.788448:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574413/real 1788574413] req@ffff920a529b3100 x1875444569836288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574413.788461:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569836288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574438 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574413.788469:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574413.788479:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574413.788483:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574414.235778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574414.811848:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574415.259787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574415.835677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574416.283778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574416.859670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574417.307740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574417.883688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574418.331798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574418.907699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574418.907737:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574418.907744:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574418.907748:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574418.907759:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11172 00000100:00080000:1.0:1788574418.907769:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574418.907777:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574418.907813:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569836416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574418.907838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574418.907842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574418.907844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574418.907848:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11172 00000100:00080000:1.0:1788574418.907852:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574418.907858:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574418.907868:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569836416:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574418.907876:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574418.907894:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569836544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574418.907906:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574418.907910:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574418.907912:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574418.907917:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11172 00000800:00000100:0.0:1788574418.907928:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574418.907930:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574418.907936:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574418.907942:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574418.907956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569836672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574418.907967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574418.907968:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574418/real 1788574418] req@ffff920a52890700 x1875444569836416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574418.907974:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574418.907979:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574418.907989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11172 00000100:00080000:1.0:1788574418.907996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574418.908003:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574418.908009:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574418.908011:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569836416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574418.908023:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569836800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574418.908029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574418.908035:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574418.908036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574418.908043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569836416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574418.908045:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574418.908049:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574418.908054:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11172 00000100:00080000:1.0:1788574418.908058:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574418.908063:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574418.908067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836544:192.168.202.157@tcp:38:kworker.0 00000008:00100000:1.0:1788574418.908068:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574418.908085:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569836928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574418.908109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574418.908126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574418/real 1788574418] req@ffff920a52893100 x1875444569836544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574418.908139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569836544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574418.908145:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574418.908148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574418.908150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574418.908154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574418.908172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574418.908183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574418/real 1788574418] req@ffff920a52891180 x1875444569836672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574418.908196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569836672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574418.908211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574418.908214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574418.908217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574418.908221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574418.908237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574418.908260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574418/real 1788574418] req@ffff920a52892300 x1875444569836800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574418.908269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569836800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574418.908274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574418.908276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574418.908278:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574418.908292:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574418.908306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574418.908314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574418/real 1788574418] req@ffff920a52892d80 x1875444569836928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574418.908326:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569836928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574443 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574418.908334:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574418.908337:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574418.908342:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569836928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574419.355773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574419.931685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574420.379768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574420.955672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574421.403791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574421.979683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574422.427751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574423.003708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574423.451740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574424.027671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574424.027709:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574424.027717:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574424.027721:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574424.027731:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11177 00000100:00080000:3.0:1788574424.027742:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574424.027750:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574424.027787:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569837056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574424.027808:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574424.027811:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574424.027813:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574424.027816:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11177 00000100:00080000:3.0:1788574424.027819:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574424.027822:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574424.027830:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574424.027839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569837056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574424.027843:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569837184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574424.027851:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574424.027854:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574424.027855:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574424.027858:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11177 00000100:00080000:3.0:1788574424.027862:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574424.027864:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574424.027868:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574424.027876:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569837312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574424.027883:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574424.027886:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574424.027887:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574424.027890:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11177 00000100:00080000:3.0:1788574424.027892:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574424.027895:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574424.027900:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574424.027902:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574424.027911:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569837440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574424.027919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574424.027921:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574424.027922:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574424.027925:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11177 00000100:00080000:3.0:1788574424.027927:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574424.027930:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574424.027934:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574424.027940:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff920a529b1f80 x1875444569837056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574424.027941:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569837568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574424.027980:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569837056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.027993:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.027998:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574424.028003:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569837056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574424.028028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574424.028048:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574424.028057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff920a529b0000 x1875444569837184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.028066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569837184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.028077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.028079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574424.028081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574424.028085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574424.028096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574424.028103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff920a529b0700 x1875444569837312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.028111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569837312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.028116:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.028118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574424.028120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574424.028123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574424.028134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574424.028140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff920a529b2300 x1875444569837440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.028148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569837440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.028152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.028154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574424.028157:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574424.028160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574424.028170:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574424.028175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574424/real 1788574424] req@ffff920a529b0380 x1875444569837568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574424.028182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569837568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574449 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574424.028187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574424.028188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574424.028190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574424.475739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574425.051708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574425.499767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574426.075668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574426.523709:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574427.099681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574427.547746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574428.123931:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574428.571784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574429.147686:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574429.147689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574429.147699:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574429.147704:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574429.147714:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11182 00000100:00080000:3.0:1788574429.147726:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574429.147735:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574429.147775:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569837696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574429.147806:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574429.147817:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574429.147819:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574429.147824:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11182 00000100:00080000:3.0:1788574429.147828:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574429.147833:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574429.147843:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574429.147857:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569837824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574429.147868:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569837696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574429.147871:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574429.147879:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574429.147882:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574429.147887:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11182 00000100:00080000:3.0:1788574429.147891:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574429.147895:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574429.147900:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574429.147913:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569837952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574429.147923:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574429.147927:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574429.147929:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574429.147933:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11182 00000800:00000100:0.0:1788574429.147939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574429.147940:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574429.147947:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574429.147953:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574429.147966:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569838080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574429.147973:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff920a529b1500 x1875444569837696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.147980:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574429.147987:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574429.147989:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574429.147994:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569837696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574429.147997:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11182 00000100:00080000:3.0:1788574429.148005:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574429.148012:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788574429.148013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574429.148018:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574429.148022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574429.148030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569837696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574429.148034:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569838208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574429.148060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574429.148084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574429.148094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff920a529b1180 x1875444569837824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.148106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569837824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.148116:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.148120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574429.148123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574429.148130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574429.148150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574429.148160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff920a529b0e00 x1875444569837952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.148169:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569837952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.148177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.148181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574429.148184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569837952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574429.148190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574429.148212:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574429.148223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff920a529b3b80 x1875444569838080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.148232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569838080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.148238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.148240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574429.148242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574429.148245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574429.148258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574429.148264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574429/real 1788574429] req@ffff920a529b0380 x1875444569838208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574429.148271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569838208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574454 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574429.148285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574429.148287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574429.148289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574429.595719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574430.171708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574430.619730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574431.195670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574431.643730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574432.219691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574432.667759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574433.243698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574433.691783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574434.267740:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574434.267750:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574434.267755:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574434.267767:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11187 00000100:00080000:1.0:1788574434.267779:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574434.267787:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574434.267794:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788574434.267832:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569838336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574434.267856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574434.267859:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574434.267860:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574434.267864:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11187 00000100:00080000:1.0:1788574434.267867:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574434.267870:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574434.267881:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574434.267890:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569838464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574434.267897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574434.267900:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574434.267902:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574434.267904:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11187 00000100:00080000:1.0:1788574434.267906:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574434.267909:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574434.267912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574434.267931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569838336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574434.267932:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569838592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574434.267957:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574434.267960:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574434.267961:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574434.267966:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11187 00000100:00080000:1.0:1788574434.267968:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574434.267971:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574434.267975:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574434.267985:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569838720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574434.267992:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574434.267994:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574434.267996:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574434.267998:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11187 00000100:00080000:1.0:1788574434.268004:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574434.268008:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574434.268012:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574434.268017:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574434.268024:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569838848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574434.268062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff920a52893800 x1875444569838336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.268083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569838336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.268092:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.268095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574434.268099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569838336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574434.268105:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574434.268121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574434.268129:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff920a52890e00 x1875444569838464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.268137:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569838464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.268142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.268144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574434.268146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574434.268167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574434.268178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574434.268185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff920a52890380 x1875444569838592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.268192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569838592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.268201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.268204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574434.268206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574434.268209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574434.268220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574434.268226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff920a52893480 x1875444569838720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.268233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569838720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.268238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.268240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574434.268242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574434.268246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574434.268255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574434.268260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574434/real 1788574434] req@ffff920a52891880 x1875444569838848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574434.268267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569838848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574459 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574434.268272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574434.268274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574434.268277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569838848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574434.715823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574435.291762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574435.739736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574436.315707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574436.763718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574437.339709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574437.787773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574438.363715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574438.811792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574439.387699:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574439.387709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574439.387711:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574439.387717:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574439.387728:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11192 00000100:00080000:1.0:1788574439.387740:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574439.387748:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574439.387793:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569838976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574439.387824:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574439.387829:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574439.387831:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574439.387835:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11192 00000100:00080000:1.0:1788574439.387840:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574439.387851:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574439.387860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569838976:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574439.387864:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574439.387882:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569839104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574439.387891:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574439.387894:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574439.387896:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574439.387899:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11192 00000100:00080000:1.0:1788574439.387902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574439.387905:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574439.387910:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574439.387919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569839232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574439.387926:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574439.387929:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574439.387930:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574439.387932:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11192 00000100:00080000:1.0:1788574439.387935:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574439.387937:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574439.387941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574439.387944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574439.387953:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569839360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574439.387961:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574439.387963:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574439.387965:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574439.387969:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11192 00000100:00080000:1.0:1788574439.387973:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574439.387975:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574439.387979:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574439.387987:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569839488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574439.387992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff920a52892a00 x1875444569838976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.388011:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569838976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.388020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.388024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574439.388027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569838976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574439.388049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574439.388073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574439.388087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff920a52891f80 x1875444569839104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.388101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569839104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.388110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.388114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574439.388116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574439.388121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574439.388137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574439.388145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff920a52891500 x1875444569839232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.388152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569839232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.388158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.388160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574439.388163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574439.388166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574439.388177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574439.388183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff920a52892680 x1875444569839360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.388190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569839360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.388195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.388197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574439.388199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574439.388202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574439.388213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574439.388219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574439/real 1788574439] req@ffff920a52890a80 x1875444569839488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574439.388227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569839488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574464 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574439.388231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574439.388233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574439.388235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574439.835756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574440.411733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574440.859758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574441.435702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574441.883766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574442.459699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574442.907761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574443.483689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574443.931743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574444.507682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574444.507717:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574444.507729:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574444.507734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574444.507747:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11197 00000100:00080000:1.0:1788574444.507759:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574444.507768:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574444.507819:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569839616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574444.507852:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574444.507856:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574444.507858:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574444.507863:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11197 00000100:00080000:1.0:1788574444.507867:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574444.507872:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574444.507884:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574444.507900:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569839744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574444.507913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569839616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574444.507925:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574444.507929:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574444.507932:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574444.507937:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11197 00000100:00080000:1.0:1788574444.507942:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574444.507947:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574444.507953:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574444.507970:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569839872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574444.507980:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574444.507984:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574444.507986:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574444.507990:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11197 00000100:00080000:1.0:1788574444.507995:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574444.507998:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574444.508003:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574444.508008:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574444.508016:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569840000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574444.508023:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574444.508026:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574444.508028:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574444.508030:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11197 00000100:00080000:1.0:1788574444.508034:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574444.508037:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574444.508041:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574444.508053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff920a52890000 x1875444569839616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574444.508054:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569840128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574444.508094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569839616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574444.508108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574444.508115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574444.508120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569839616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574444.508152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574444.508174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574444.508184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff920a52893b80 x1875444569839744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574444.508193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569839744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574444.508199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574444.508202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574444.508204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574444.508208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574444.508219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574444.508224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff920a52890a80 x1875444569839872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574444.508232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569839872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574444.508237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574444.508239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574444.508241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569839872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574444.508244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574444.508255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574444.508260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff920a52892680 x1875444569840000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574444.508268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569840000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574444.508274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574444.508275:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574444.508277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574444.508280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574444.508290:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574444.508296:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574444/real 1788574444] req@ffff920a52891500 x1875444569840128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574444.508302:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569840128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574469 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574444.508307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574444.508309:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574444.508311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574444.955760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574445.531715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574445.979769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574446.555697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574447.003770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574447.579702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574448.027822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574448.603672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574449.051706:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574449.627718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574449.627751:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574449.627759:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574449.627764:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574449.627775:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11203 00000100:00080000:1.0:1788574449.627787:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574449.627794:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574449.627840:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569840256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574449.627870:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574449.627873:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574449.627878:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574449.627881:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11203 00000100:00080000:1.0:1788574449.627884:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574449.627887:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574449.627898:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574449.627908:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569840384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574449.627917:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574449.627922:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574449.627924:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574449.627926:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569840256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574449.627929:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11203 00000100:00080000:1.0:1788574449.627936:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574449.627938:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574449.627942:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574449.627950:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569840512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574449.627956:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574449.627959:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574449.627960:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574449.627963:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11203 00000100:00080000:1.0:1788574449.627965:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574449.627967:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574449.627970:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574449.627978:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569840640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574449.627983:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574449.627985:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574449.627986:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574449.627988:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11203 00000100:00080000:1.0:1788574449.627992:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574449.627995:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574449.627999:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574449.628005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574449.628009:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569840768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574449.628047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574449/real 1788574449] req@ffff920a52891f80 x1875444569840256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574449.628064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569840256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574449.628074:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574449.628077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574449.628080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569840256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574449.628104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574449.628124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574449.628134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574449/real 1788574449] req@ffff920a52892a00 x1875444569840384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574449.628148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569840384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574449.628154:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574449.628156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574449.628158:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574449.628161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574449.628174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574449.628180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574449/real 1788574449] req@ffff920a52891880 x1875444569840512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574449.628189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569840512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574449.628195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574449.628199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574449.628203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574449.628208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574449.628227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574449.628238:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574449/real 1788574449] req@ffff920a52893480 x1875444569840640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574449.628247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569840640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574449.628254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574449.628257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574449.628259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574449.628264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574449.628287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574449.628308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574449/real 1788574449] req@ffff920a52890380 x1875444569840768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574449.628317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569840768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574474 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574449.628323:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574449.628326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574449.628329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569840768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574450.075791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574450.651644:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574451.099728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574451.675696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574452.123853:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574452.699695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574453.147791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574453.723744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574454.171779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574454.747713:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574454.747721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574454.747726:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574454.747733:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574454.747745:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11208 00000100:00080000:1.0:1788574454.747756:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574454.747763:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574454.747803:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569840896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574454.747838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574454.747842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574454.747843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574454.747848:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11208 00000100:00080000:1.0:1788574454.747852:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574454.747856:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574454.747866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574454.747879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569840896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574454.747882:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569841024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574454.747901:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574454.747905:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574454.747906:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574454.747910:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11208 00000100:00080000:1.0:1788574454.747913:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574454.747916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574454.747920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574454.747929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569841152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574454.747936:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574454.747940:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574454.747941:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574454.747944:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11208 00000100:00080000:1.0:1788574454.747949:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574454.747952:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574454.747955:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574454.747962:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569841280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574454.747970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574454.747979:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574454.747982:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574454.747984:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574454.747989:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11208 00000100:00080000:1.0:1788574454.747995:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574454.748000:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574454.748006:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574454.748021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574454/real 1788574454] req@ffff920a52890e00 x1875444569840896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574454.748026:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569841408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574454.748053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569840896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574454.748066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574454.748070:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574454.748077:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569840896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574454.748110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574454.748128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574454.748143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574454/real 1788574454] req@ffff920a52893800 x1875444569841024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574454.748152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569841024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574454.748158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574454.748161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574454.748163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574454.748168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574454.748180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574454.748187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574454/real 1788574454] req@ffff920a52892d80 x1875444569841152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574454.748200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569841152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574454.748205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574454.748207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574454.748209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574454.748212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574454.748224:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574454.748231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574454/real 1788574454] req@ffff920a52892300 x1875444569841280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574454.748238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569841280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574454.748243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574454.748245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574454.748247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574454.748251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574454.748261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574454.748266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574454/real 1788574454] req@ffff920a52891180 x1875444569841408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574454.748273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569841408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574479 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574454.748283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574454.748285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574454.748287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788574455.195804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574455.771682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574456.219792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574456.795727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574457.243779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574457.819769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574458.267790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574458.843739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574459.291796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574459.867771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574459.867779:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574459.867794:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574459.867799:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574459.867814:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11213 00000100:00080000:0.0:1788574459.867829:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574459.867839:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574459.867880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569841536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574459.867910:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574459.867913:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574459.867915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574459.867918:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11213 00000100:00080000:0.0:1788574459.867921:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574459.867924:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574459.867934:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574459.867950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569841664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574459.867961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574459.867962:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569841536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574459.867975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574459.867979:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574459.867986:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11213 00000100:00080000:0.0:1788574459.867991:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574459.867995:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574459.868002:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574459.868018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569841792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574459.868027:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574459.868029:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574459.868032:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574459.868035:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11213 00000100:00080000:0.0:1788574459.868038:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574459.868042:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574459.868046:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574459.868055:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569841920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574459.868066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574459.868068:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574459.868072:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574459.868074:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574459.868077:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11213 00000100:00080000:0.0:1788574459.868085:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574459.868088:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574459.868093:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574459.868100:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569842048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574459.868130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574459/real 1788574459] req@ffff920a48350700 x1875444569841536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574459.868155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569841536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574459.868169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574459.868175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574459.868181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569841536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574459.868212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574459.868240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574459.868254:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574459/real 1788574459] req@ffff920a48350380 x1875444569841664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574459.868269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569841664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574459.868288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574459.868292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574459.868296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574459.868303:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574459.868324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574459.868336:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574459/real 1788574459] req@ffff920a48353480 x1875444569841792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574459.868348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569841792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574459.868358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574459.868362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574459.868365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574459.868373:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574459.868397:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574459.868411:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574459/real 1788574459] req@ffff920a48351500 x1875444569841920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574459.868425:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569841920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574459.868439:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574459.868443:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574459.868447:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569841920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574459.868453:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574459.868474:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574459.868485:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574459/real 1788574459] req@ffff920a48353b80 x1875444569842048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574459.868498:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569842048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574484 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574459.868507:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574459.868511:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574459.868514:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574460.315802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574460.891785:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574461.339807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574461.915743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574462.363768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574462.939695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788574463.387846:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574463.963773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574464.411800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574464.987678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574464.987750:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574464.987769:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574464.987773:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574464.987782:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11218 00000100:00080000:1.0:1788574464.987789:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574464.987795:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574464.987825:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569842176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574464.987846:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574464.987849:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574464.987851:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574464.987854:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11218 00000100:00080000:1.0:1788574464.987857:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574464.987859:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574464.987867:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574464.987876:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569842304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574464.987885:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574464.987888:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574464.987889:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574464.987894:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569842176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574464.987896:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11218 00000100:00080000:1.0:1788574464.987902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574464.987906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574464.987910:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574464.987919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569842432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574464.987925:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574464.987928:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574464.987930:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574464.987933:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11218 00000100:00080000:1.0:1788574464.987936:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574464.987939:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574464.987942:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574464.987948:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569842560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574464.987956:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574464.987956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574464.987962:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574464.987964:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574464.987967:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11218 00000100:00080000:1.0:1788574464.987969:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574464.987972:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574464.987975:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574464.987983:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569842688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574464.987994:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574464/real 1788574464] req@ffff920a52893100 x1875444569842176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574464.988010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569842176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574464.988018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574464.988020:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574464.988023:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569842176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574464.988041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574464.988056:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574464.988064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574464/real 1788574464] req@ffff920a52890700 x1875444569842304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574464.988072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569842304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574464.988077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574464.988079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574464.988081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574464.988084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574464.988094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574464.988100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574464/real 1788574464] req@ffff920a52891180 x1875444569842432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574464.988106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569842432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574464.988116:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574464.988118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574464.988120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574464.988123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574464.988133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574464.988138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574464/real 1788574464] req@ffff920a52892300 x1875444569842560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574464.988145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569842560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574464.988150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574464.988152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574464.988154:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574464.988157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574464.988167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574464.988176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574464/real 1788574464] req@ffff920a52892d80 x1875444569842688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574464.988187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569842688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574489 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574464.988195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574464.988198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574464.988203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574465.435798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574466.011708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574466.459788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574467.035670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574467.483753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574468.059703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574468.507743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574469.083717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574469.531820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574470.107689:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574470.107690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574470.107701:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574470.107707:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.107719:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11223 00000100:00080000:1.0:1788574470.107732:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.107741:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574470.107787:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569842816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.107827:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.107831:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574470.107834:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.107838:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11223 00000100:00080000:1.0:1788574470.107842:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.107848:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574470.107859:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574470.107872:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569842944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574470.107874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569842816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574470.107885:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.107889:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574470.107890:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.107893:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11223 00000100:00080000:1.0:1788574470.107896:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.107899:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574470.107903:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574470.107912:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569843072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.107919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.107921:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574470.107923:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.107925:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11223 00000100:00080000:1.0:1788574470.107927:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.107929:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574470.107933:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574470.107940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569843200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574470.107946:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574470.107948:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574470.107950:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574470.107952:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11223 00000100:00080000:1.0:1788574470.107954:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574470.107956:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574470.107959:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574470.107964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574470.107972:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569843328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574470.108005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff920a52893800 x1875444569842816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574470.108021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569842816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574470.108030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574470.108034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574470.108037:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569842816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574470.108060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574470.108076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574470.108085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff920a52890e00 x1875444569842944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574470.108093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569842944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574470.108099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574470.108101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574470.108103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569842944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574470.108106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574470.108117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574470.108124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff920a52890380 x1875444569843072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574470.108131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569843072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574470.108137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574470.108138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574470.108140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574470.108144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574470.108155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574470.108160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff920a52893480 x1875444569843200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574470.108168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569843200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574470.108173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574470.108175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574470.108176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574470.108180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574470.108190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574470.108195:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574470/real 1788574470] req@ffff920a52891880 x1875444569843328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574470.108201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569843328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574495 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574470.108207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574470.108210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574470.108212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574470.555779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574471.131757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574471.579789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574472.155673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574472.603751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574473.179730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574473.627806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574474.203737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574474.651780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574475.227729:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574475.227735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574475.227737:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574475.227743:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.227754:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11228 00000100:00080000:1.0:1788574475.227765:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.227772:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574475.227806:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569843456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.227827:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.227830:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574475.227831:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.227834:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11228 00000100:00080000:1.0:1788574475.227836:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.227839:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574475.227849:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574475.227858:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569843584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.227884:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574475.227886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569843456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574475.227887:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574475.227893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.227907:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11228 00000100:00080000:1.0:1788574475.227910:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.227914:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574475.227920:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574475.227939:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569843712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.227951:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.227955:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574475.227956:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.227962:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11228 00000100:00080000:1.0:1788574475.227968:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.227973:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574475.227980:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574475.227994:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569843840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574475.228007:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574475.228010:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574475.228013:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574475.228017:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11228 00000100:00100000:2.0:1788574475.228021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843584:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788574475.228023:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574475.228034:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574475.228039:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574475.228055:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569843968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574475.228061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574475.228092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574475.228116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574475.675773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574476.251731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574476.699811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574477.275727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574477.723804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574478.299697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574478.299991:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574478.300047:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574478.300070:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574478.300123:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574478.300138:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574478.300151:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574478.300164:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574478.300190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574478] req@ffff920a52891f80 x1875444569843584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574478.300222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569843584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574478.300236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574478.300242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574478.300246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574478.300255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574478] req@ffff920a52891500 x1875444569843712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574478.300270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569843712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574478.300279:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574478.300283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574478.300286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574478.300295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574478] req@ffff920a52892680 x1875444569843840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574478.300308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569843840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574478.300324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574478.300328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574478.300333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574478.300342:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574478] req@ffff920a52892a00 x1875444569843456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574478.300356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569843456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574478.300365:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574478.300369:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574478.300374:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569843456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574478.300421:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574475/real 1788574478] req@ffff920a52890a80 x1875444569843968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574478.300440:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569843968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574500 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574478.300450:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574478.300454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574478.300459:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569843968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574478.747766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574479.323718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574479.771794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574480.347737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574480.347817:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574480.347825:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574480.347831:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574480.347843:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11233 00000100:00080000:0.0:1788574480.347857:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574480.347866:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574480.347914:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569844096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574480.347984:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574480.347989:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574480.347991:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574480.347996:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11233 00000100:00080000:0.0:1788574480.348001:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574480.348006:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574480.348025:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574480.348043:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569844224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574480.348049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569844096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574480.348070:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574480.348076:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574480.348078:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574480.348083:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11233 00000100:00080000:0.0:1788574480.348088:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574480.348093:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574480.348099:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574480.348114:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569844352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574480.348125:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574480.348128:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574480.348130:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574480.348135:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11233 00000100:00080000:0.0:1788574480.348139:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574480.348144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574480.348148:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574480.348157:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574480.348172:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569844480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574480.348184:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574480.348188:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574480.348191:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574480.348198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff920a48350a80 x1875444569844096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574480.348210:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11233 00000100:00080000:0.0:1788574480.348219:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574480.348227:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574480.348233:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574480.348239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569844096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574480.348251:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569844608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574480.348254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.348267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574480.348276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569844096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574480.348315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574480.348341:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574480.348355:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff920a48351c00 x1875444569844224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.348367:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569844224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.348375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.348378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574480.348382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574480.348387:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574480.348408:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574480.348430:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff920a48353800 x1875444569844352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.348442:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569844352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.348456:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.348459:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574480.348462:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574480.348469:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574480.348492:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574480.348504:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff920a48353b80 x1875444569844480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.348515:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569844480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.348523:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.348527:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574480.348530:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574480.348537:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574480.348672:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574480.348688:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574480/real 1788574480] req@ffff920a48351500 x1875444569844608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574480.348702:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569844608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574505 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574480.348711:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574480.348714:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574480.348718:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574480.795752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574481.371751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574481.819819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574482.395702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574482.843772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574483.419728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574483.867808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574484.443657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574484.891758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574485.467704:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.467714:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574485.467718:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.467727:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11238 00000100:00100000:2.0:1788574485.467733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574485.467737:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.467747:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574485.467794:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569844736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574485.467826:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.467831:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574485.467833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.467841:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11238 00000100:00080000:1.0:1788574485.467848:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.467856:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574485.467866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569844736:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574485.467868:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574485.467888:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569844864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574485.467902:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.467906:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574485.467909:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.467914:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11238 00000100:00080000:1.0:1788574485.467929:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.467934:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574485.467941:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574485.467946:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574485.467958:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569844992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574485.467968:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574485.467971:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574485.467973:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574485.467977:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11238 00000100:00080000:1.0:1788574485.467996:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.468001:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574485.468006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff920a52893b80 x1875444569844736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788574485.468010:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574485.468029:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569845120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574485.468034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569844736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574485.468055:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574485.468059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574485.468059:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574485.468064:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574485.468068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788574485.468071:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11238 00000100:00080000:1.0:1788574485.468076:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574485.468082:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574485.468087:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574485.468088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569844736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574485.468100:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569845248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574485.468129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574485.468156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574485.468180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff920a52890000 x1875444569844864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574485.468204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569844864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574485.468214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574485.468218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574485.468223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574485.468230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574485.468254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574485.468268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff920a52890a80 x1875444569844992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574485.468278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569844992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574485.468284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574485.468288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574485.468290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569844992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574485.468297:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574485.468312:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574485.468322:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff920a52892a00 x1875444569845120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574485.468334:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569845120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574485.468344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574485.468348:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574485.468351:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574485.468355:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574485.468370:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574485.468383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574485/real 1788574485] req@ffff920a52892680 x1875444569845248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574485.468394:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569845248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574510 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574485.468400:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574485.468403:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574485.468406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574485.915734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574486.491707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574486.939820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574487.515719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574487.963725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574488.539744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574488.987777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574489.563680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574490.011764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574490.587748:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574490.587758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574490.587761:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574490.587767:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.587781:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11243 00000100:00080000:1.0:1788574490.587793:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.587803:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574490.587851:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569845376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.587876:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.587878:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574490.587880:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.587883:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11243 00000100:00080000:1.0:1788574490.587886:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.587889:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574490.587899:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574490.587914:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569845376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574490.587914:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569845504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.587929:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.587931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574490.587932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.587935:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11243 00000100:00080000:1.0:1788574490.587938:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.587940:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574490.587945:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574490.587955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569845632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.587962:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574490.587964:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574490.587966:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.587968:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11243 00000100:00080000:1.0:1788574490.587979:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.587982:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574490.587986:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574490.587994:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569845760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574490.588001:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574490.588006:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574490.588006:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574490.588010:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574490.588014:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11243 00000100:00080000:1.0:1788574490.588017:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574490.588019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574490.588021:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574490.588029:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569845888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574490.588071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff920a52891500 x1875444569845376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574490.588090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569845376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574490.588101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574490.588106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574490.588111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569845376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574490.588139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574490.588160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574490.588170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff920a52891f80 x1875444569845504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574490.588178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569845504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574490.588184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574490.588186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574490.588190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574490.588196:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574490.588216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574490.588228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff920a52891880 x1875444569845632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574490.588241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569845632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574490.588249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574490.588251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574490.588253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574490.588258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574490.588275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574490.588282:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff920a52893480 x1875444569845760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574490.588290:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569845760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574490.588295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574490.588297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574490.588299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574490.588302:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574490.588313:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574490.588319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574490/real 1788574490] req@ffff920a52890380 x1875444569845888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574490.588326:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569845888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574515 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574490.588332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574490.588333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574490.588335:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569845888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574491.035789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574491.611681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574492.059782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574492.635689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574493.083794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574493.659697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574494.107766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574494.683690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574495.131764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574495.707704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574495.707711:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574495.707720:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574495.707724:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574495.707736:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11249 00000100:00080000:1.0:1788574495.707748:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574495.707756:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574495.707816:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569846016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574495.707847:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574495.707851:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574495.707853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574495.707858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11249 00000100:00080000:1.0:1788574495.707863:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574495.707868:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574495.707881:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574495.707888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569846016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574495.707903:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569846144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574495.707916:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574495.707920:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574495.707923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574495.707928:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11249 00000100:00080000:1.0:1788574495.707944:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574495.707949:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574495.707956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788574495.707956:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574495.707977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569846272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574495.707989:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574495.707993:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574495.707997:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574495.707998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574495/real 1788574495] req@ffff920a52890e00 x1875444569846016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574495.708011:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11249 00000100:00080000:1.0:1788574495.708020:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574495.708027:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574495.708035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569846016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788574495.708035:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574495.708056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788574495.708057:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569846400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574495.708066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574495.708087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569846016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574495.708094:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574495.708098:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574495.708100:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574495.708105:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11249 00000100:00080000:1.0:1788574495.708108:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574495.708112:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574495.708118:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574495.708119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574495.708135:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569846528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574495.708160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574495.708177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574495/real 1788574495] req@ffff920a52893800 x1875444569846144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574495.708187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569846144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574495.708193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574495.708196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574495.708198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574495.708203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574495.708218:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574495.708225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574495/real 1788574495] req@ffff920a52892d80 x1875444569846272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574495.708232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569846272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574495.708242:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574495.708244:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574495.708246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574495.708250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574495.708261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574495.708267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574495/real 1788574495] req@ffff920a52892300 x1875444569846400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574495.708273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569846400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574495.708279:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574495.708281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574495.708283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574495.708296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574495.708307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574495.708312:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574495/real 1788574495] req@ffff920a52891180 x1875444569846528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574495.708328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569846528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574520 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574495.708334:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574495.708336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574495.708338:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574496.155781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574496.731704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574497.179816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574497.755676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574498.203797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574498.779760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574499.227797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574499.803706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574500.251782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574500.827709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574500.827717:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574500.827726:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574500.827731:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574500.827753:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11254 00000100:00080000:0.0:1788574500.827767:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574500.827777:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574500.827820:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569846656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574500.827844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574500.827847:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574500.827849:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574500.827853:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11254 00000100:00080000:0.0:1788574500.827858:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574500.827863:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574500.827876:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574500.827881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569846656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574500.827895:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569846784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574500.827908:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574500.827911:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574500.827913:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574500.827918:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11254 00000100:00080000:0.0:1788574500.827922:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574500.827926:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574500.827931:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574500.827940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569846912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574500.827947:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574500.827950:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574500.827951:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574500.827956:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11254 00000800:00000100:3.0:1788574500.827957:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574500.827960:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574500.827963:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574500.827967:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574500.827978:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569847040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574500.827985:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574500.827988:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574500.827989:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574500.827992:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11254 00000100:00080000:0.0:1788574500.827995:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574500.827998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574500/real 1788574500] req@ffff920a48353480 x1875444569846656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574500.828000:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574500.828007:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574500.828023:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569846656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574500.828027:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569847168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574500.828036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574500.828044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574500.828050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569846656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574500.828073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574500.828090:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574500.828115:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574500/real 1788574500] req@ffff920a48350380 x1875444569846784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574500.828125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569846784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574500.828132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574500.828135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574500.828139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574500.828146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574500.828167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574500.828180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574500/real 1788574500] req@ffff920a48350700 x1875444569846912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574500.828193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569846912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574500.828201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574500.828205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574500.828209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569846912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574500.828215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574500.828237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574500.828247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574500/real 1788574500] req@ffff920a48351500 x1875444569847040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574500.828259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569847040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574500.828267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574500.828271:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574500.828274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574500.828281:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574500.828300:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574500.828311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574500/real 1788574500] req@ffff920a48353b80 x1875444569847168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574500.828323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569847168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574525 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574500.828332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574500.828336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574500.828340:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574501.275828:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574501.851676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574502.299758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574502.875693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574503.323792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574503.899648:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574504.347751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574504.923678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574505.371790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574505.947758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574505.947776:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574505.947785:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574505.947789:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574505.947801:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11259 00000100:00080000:0.0:1788574505.947810:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574505.947818:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574505.947858:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569847296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574505.947884:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574505.947889:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574505.947892:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574505.947897:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11259 00000100:00080000:0.0:1788574505.947902:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574505.947907:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574505.947917:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574505.947925:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569847296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574505.947933:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569847424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574505.947943:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574505.947946:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574505.947947:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574505.947952:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11259 00000100:00080000:0.0:1788574505.947955:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574505.947958:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574505.947962:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574505.947975:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569847552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574505.947983:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574505.947986:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574505.947987:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574505.947990:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11259 00000100:00080000:0.0:1788574505.947992:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574505.947994:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574505.947999:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574505.948008:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569847680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574505.948017:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574505.948028:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574505.948032:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574505.948034:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574505.948039:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11259 00000100:00080000:0.0:1788574505.948044:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574505.948049:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574505.948053:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574505.948071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574505/real 1788574505] req@ffff920a48353800 x1875444569847296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574505.948075:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569847808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574505.948148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569847296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574505.948161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574505.948164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574505.948168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569847296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574505.948195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574505.948216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574505.948226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574505/real 1788574505] req@ffff920a48351c00 x1875444569847424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574505.948234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569847424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574505.948241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574505.948243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574505.948245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574505.948251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574505.948262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574505.948268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574505/real 1788574505] req@ffff920a48350a80 x1875444569847552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574505.948276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569847552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574505.948281:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574505.948283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574505.948285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574505.948288:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574505.948299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574505.948305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574505/real 1788574505] req@ffff920a48353b80 x1875444569847680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574505.948312:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569847680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574505.948317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574505.948319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574505.948321:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574505.948325:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574505.948335:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574505.948340:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574505/real 1788574505] req@ffff920a48351500 x1875444569847808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574505.948348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569847808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574530 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574505.948353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574505.948355:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574505.948356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569847808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574506.395856:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574506.971714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574507.419772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574507.995757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574508.443868:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574509.019685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574509.467795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574510.043696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574510.491788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574511.067723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574511.067730:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574511.067746:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574511.067751:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574511.067763:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11264 00000100:00080000:3.0:1788574511.067775:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574511.067783:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574511.067829:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569847936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574511.067856:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574511.067859:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574511.067860:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574511.067863:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11264 00000100:00080000:3.0:1788574511.067866:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574511.067869:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574511.067879:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574511.067888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569848064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574511.067899:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569847936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574511.067903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574511.067907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574511.067908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574511.067911:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11264 00000100:00080000:3.0:1788574511.067913:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574511.067916:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574511.067919:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574511.067929:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569848192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574511.067936:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574511.067939:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574511.067940:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574511.067942:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11264 00000100:00080000:3.0:1788574511.067944:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574511.067947:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574511.067951:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574511.067957:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569848320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574511.067964:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574511.067967:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574511.067968:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574511.067971:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11264 00000800:00000100:0.0:1788574511.067975:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574511.067976:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574511.067981:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574511.067985:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574511.067993:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569848448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574511.068018:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff920a529b2300 x1875444569847936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574511.068034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569847936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574511.068043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574511.068046:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574511.068049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569847936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574511.068072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574511.068089:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574511.068097:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff920a529b0700 x1875444569848064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574511.068106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569848064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574511.068115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574511.068119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574511.068123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574511.068129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574511.068151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574511.068179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff920a529b0000 x1875444569848192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574511.068188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569848192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574511.068194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574511.068196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574511.068198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574511.068203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574511.068219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574511.068226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff920a529b1f80 x1875444569848320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574511.068233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569848320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574511.068238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574511.068240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574511.068241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574511.068245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574511.068256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574511.068262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574511/real 1788574511] req@ffff920a529b3100 x1875444569848448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574511.068273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569848448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574511.068278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574511.068280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574511.068282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574511.515792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574512.091719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574512.539778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574513.115743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574513.563757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574514.139709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574514.587768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574515.163700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574515.611736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574516.187690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574516.187701:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574516.187710:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574516.187715:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574516.187732:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11269 00000100:00080000:0.0:1788574516.187743:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574516.187750:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574516.187789:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569848576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574516.187814:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574516.187819:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574516.187821:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574516.187826:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11269 00000100:00080000:0.0:1788574516.187832:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574516.187838:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574516.187847:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569848576:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574516.187850:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574516.187868:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569848704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574516.187877:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574516.187879:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574516.187881:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574516.187884:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11269 00000100:00080000:0.0:1788574516.187888:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574516.187891:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574516.187895:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574516.187902:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569848832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574516.187909:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574516.187912:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574516.187914:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574516.187915:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574516.187921:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11269 00000100:00080000:0.0:1788574516.187924:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574516.187926:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574516.187930:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574516.187939:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569848960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574516.187946:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574516.187950:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788574516.187950:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff920a48350700 x1875444569848576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574516.187954:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574516.187962:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11269 00000100:00080000:0.0:1788574516.187966:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574516.187969:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574516.187973:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574516.187974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569848576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574516.187983:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569849088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574516.187985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.187993:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574516.187999:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569848576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574516.188018:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.188040:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574516.188054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff920a48350380 x1875444569848704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.188068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569848704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.188077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.188081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574516.188085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574516.188089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574516.188105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574516.188114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff920a48353480 x1875444569848832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.188130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569848832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.188141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.188147:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574516.188149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574516.188154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574516.188166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574516.188174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff920a48351500 x1875444569848960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.188181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569848960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.188199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.188201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574516.188203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569848960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574516.188207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574516.188222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574516.188234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574516/real 1788574516] req@ffff920a48353b80 x1875444569849088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574516.188246:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569849088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574516.188254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574516.188258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574516.188261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574516.635739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574517.211714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574517.659745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574518.235771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574518.683788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574519.259667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574519.707737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574520.283692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574520.731793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574521.307667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574521.307677:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574521.307685:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574521.307690:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574521.307701:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11274 00000100:00080000:1.0:1788574521.307712:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574521.307719:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574521.307758:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569849216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574521.307784:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574521.307789:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574521.307791:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574521.307797:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11274 00000100:00080000:1.0:1788574521.307802:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574521.307807:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574521.307813:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569849216:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574521.307821:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574521.307839:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569849344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574521.307858:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574521.307863:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574521.307865:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574521.307870:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11274 00000100:00080000:1.0:1788574521.307875:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574521.307879:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574521.307884:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574521.307891:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574521.307907:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569849472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574521.307919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574521.307924:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff920a52890700 x1875444569849216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574521.307925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574521.307929:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574521.307938:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11274 00000100:00080000:1.0:1788574521.307946:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574521.307951:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574521.307957:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574521.307958:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569849216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574521.307976:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569849600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574521.307976:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574521.307987:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574521.307990:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574521.307992:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574521.307996:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11274 00000100:00080000:1.0:1788574521.308000:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574521.308004:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574521.308009:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574521.308019:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569849728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574521.308056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574521.308062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569849216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574521.308085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574521.308112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574521.308126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff920a52893100 x1875444569849344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.308287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569849344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.308305:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.308310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574521.308315:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574521.308324:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574521.308399:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574521.308413:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff920a52891180 x1875444569849472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.308427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569849472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.308435:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.308439:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574521.308443:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574521.308449:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574521.308471:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574521.308483:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff920a52892300 x1875444569849600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.308496:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569849600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.308504:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.308508:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574521.308512:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574521.308518:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574521.308539:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574521.308551:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574521/real 1788574521] req@ffff920a52892d80 x1875444569849728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574521.308563:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569849728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574546 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574521.308572:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574521.308575:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574521.308579:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574521.755789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574522.331683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574522.779803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574523.355647:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574523.803726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574524.379672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574524.827746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574525.403725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574525.851808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574526.427682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574526.427714:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574526.427725:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574526.427730:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574526.427741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11279 00000100:00080000:1.0:1788574526.427753:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574526.427760:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574526.427794:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569849856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574526.427815:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574526.427818:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574526.427820:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574526.427823:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11279 00000100:00080000:1.0:1788574526.427826:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574526.427830:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574526.427838:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574526.427845:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569849856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574526.427848:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569849984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574526.427863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574526.427866:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574526.427867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574526.427870:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11279 00000100:00080000:1.0:1788574526.427872:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574526.427874:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574526.427877:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574526.427886:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569850112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574526.427892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574526.427895:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574526.427896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574526.427898:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11279 00000100:00080000:1.0:1788574526.427901:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574526.427905:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788574526.427906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788574526.427909:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574526.427918:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569850240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574526.427924:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574526.427927:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574526.427928:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574526.427930:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11279 00000100:00080000:1.0:1788574526.427934:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574526.427936:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574526.427939:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574526.427941:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574526] req@ffff920a52893800 x1875444569849856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574526.427950:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569850368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574526.427959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569849856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574526.427969:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574526.427973:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574526.427976:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569849856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574526.427995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574526.428010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574526.428019:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574526] req@ffff920a52890e00 x1875444569849984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574526.428028:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569849984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574526.428033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574526.428035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574526.428038:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569849984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574526.428041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574526.428052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574526.428057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574526] req@ffff920a52890380 x1875444569850112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574526.428064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569850112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574526.428069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574526.428071:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574526.428073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574526.428076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574526.428086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574526.428091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574526] req@ffff920a52893480 x1875444569850240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574526.428098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569850240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574526.428104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574526.428107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574526.428109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574526.428112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574526.428121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574526.428127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574526/real 1788574526] req@ffff920a52891880 x1875444569850368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574526.428133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569850368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574551 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574526.428138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574526.428140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574526.428142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574526.875762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574527.451744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574527.899797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574528.475770:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574528.923772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574529.499748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574529.947747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574530.523709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574530.971777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574531.547743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574531.547748:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574531.547763:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574531.547768:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574531.547781:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11284 00000100:00080000:3.0:1788574531.547793:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574531.547803:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574531.547846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569850496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574531.547881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574531.547885:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574531.547886:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574531.547890:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11284 00000100:00080000:3.0:1788574531.547894:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574531.547897:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574531.547906:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574531.547917:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569850624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574531.547926:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574531.547929:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574531.547931:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574531.547935:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11284 00000100:00080000:3.0:1788574531.547937:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574531.547940:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574531.547946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574531.547947:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569850496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574531.547956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569850752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574531.547963:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574531.547966:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574531.547967:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574531.547970:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11284 00000100:00080000:3.0:1788574531.547972:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574531.547974:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574531.547979:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574531.547988:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569850880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574531.547995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574531.547997:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574531.547999:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574531.548001:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11284 00000100:00080000:3.0:1788574531.548004:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574531.548006:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574531.548009:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574531.548017:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569851008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574531.548023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574531.548066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff920a529b2d80 x1875444569850496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574531.548085:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569850496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574531.548097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574531.548102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574531.548106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569850496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574531.548123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574531.548148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574531.548161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff920a529b3480 x1875444569850624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574531.548174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569850624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574531.548184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574531.548187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574531.548191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574531.548224:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574531.548243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574531.548254:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff920a529b2a00 x1875444569850752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574531.548267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569850752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574531.548290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574531.548294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574531.548298:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574531.548305:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574531.548325:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574531.548337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff920a529b1c00 x1875444569850880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574531.548348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569850880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574531.548357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574531.548360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574531.548363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569850880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574531.548369:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574531.548386:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574531.548395:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574531/real 1788574531] req@ffff920a529b3100 x1875444569851008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574531.548406:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569851008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574556 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574531.548414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574531.548418:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574531.548421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574531.995746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574532.571745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574533.019753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574533.595738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574534.043758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574534.619714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574535.067773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574535.643766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574536.091790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574536.667712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574536.667717:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574536.667726:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574536.667730:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574536.667742:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11290 00000100:00080000:0.0:1788574536.667754:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574536.667762:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574536.667804:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569851136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574536.667834:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574536.667837:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574536.667838:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574536.667841:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11290 00000100:00080000:0.0:1788574536.667844:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574536.667846:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574536.667857:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574536.667868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569851264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574536.667879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569851136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574536.667882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574536.667892:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574536.667897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574536.667903:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11290 00000100:00080000:0.0:1788574536.667905:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574536.667907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574536.667910:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574536.667921:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569851392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574536.667928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574536.667931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574536.667933:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574536.667936:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11290 00000100:00080000:0.0:1788574536.667939:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574536.667942:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574536.667946:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574536.667955:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569851520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574536.667961:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574536.667964:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574536.667965:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574536.667968:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11290 00000100:00080000:0.0:1788574536.667971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574536.667974:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788574536.667985:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788574536.667987:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574536.668001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569851648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574536.668034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574536/real 1788574536] req@ffff920a48350a80 x1875444569851136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574536.668049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569851136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574536.668059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574536.668062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574536.668065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569851136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574536.668086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574536.668103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574536.668112:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574536/real 1788574536] req@ffff920a48351c00 x1875444569851264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574536.668120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569851264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574536.668125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574536.668127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574536.668129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574536.668134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574536.668145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574536.668151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574536/real 1788574536] req@ffff920a48353800 x1875444569851392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574536.668159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569851392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574536.668164:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574536.668165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574536.668167:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574536.668171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574536.668181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574536.668187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574536/real 1788574536] req@ffff920a48353b80 x1875444569851520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574536.668194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569851520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574536.668199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574536.668201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574536.668203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574536.668206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574536.668215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574536.668221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574536/real 1788574536] req@ffff920a48351500 x1875444569851648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574536.668227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569851648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574561 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574536.668232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574536.668234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574536.668236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574537.115827:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574537.691696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574538.139750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574538.715690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574539.163778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574539.739717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574540.187794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574540.763719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574541.211751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574541.787696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574541.787701:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574541.787712:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574541.787717:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574541.787732:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11295 00000100:00080000:0.0:1788574541.787746:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574541.787755:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574541.787806:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569851776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574541.787848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574541.787852:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574541.787854:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574541.787859:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11295 00000100:00080000:0.0:1788574541.787864:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574541.787868:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574541.787881:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574541.787898:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569851904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574541.787906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569851776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574541.787913:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574541.787928:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574541.787930:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574541.787936:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11295 00000100:00080000:0.0:1788574541.787941:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574541.787945:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574541.787952:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574541.787965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569852032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574541.787976:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574541.787980:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574541.787982:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574541.787987:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11295 00000100:00080000:0.0:1788574541.787991:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574541.787996:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574541.788002:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574541.788018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569852160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574541.788030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851904:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574541.788031:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574541.788040:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574541.788045:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574541.788052:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11295 00000100:00080000:0.0:1788574541.788059:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574541.788065:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574541.788073:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574541.788088:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569852288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574541.788092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574541.788141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574541.788170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574542.235747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574542.811749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574543.259816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574543.835691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574544.283741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574544.859715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788574544.859880:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788574544.859923:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574544.859940:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574544.859983:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574544.859994:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574544.860008:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574544.860024:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574544.860047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574541/real 1788574544] req@ffff920a48351500 x1875444569852160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574544.860077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569852160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574544.860090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574544.860095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574544.860100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574544.860109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574541/real 1788574544] req@ffff920a48353b80 x1875444569852288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574544.860124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569852288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574544.860133:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574544.860137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574544.860141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852288:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574544.860149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574541/real 1788574544] req@ffff920a48350380 x1875444569851904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574544.860163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569851904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574544.860172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574544.860177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574544.860180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569851904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574544.860188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574541/real 1788574544] req@ffff920a48350700 x1875444569852032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574544.860201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569852032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574544.860215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574544.860220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574544.860223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574544.860231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574541/real 1788574544] req@ffff920a48353480 x1875444569851776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574544.860244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569851776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574566 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574544.860252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574544.860256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574544.860259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569851776:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788574545.307819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574545.883694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574546.331757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574546.907723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574546.907755:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574546.907761:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574546.907765:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574546.907776:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11300 00000100:00080000:0.0:1788574546.907787:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574546.907796:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574546.907845:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569852416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574546.907878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574546.907882:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574546.907885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574546.907890:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11300 00000100:00080000:0.0:1788574546.907894:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574546.907899:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574546.907911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574546.907927:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569852544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574546.907940:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574546.907944:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574546.907947:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574546.907952:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11300 00000100:00080000:0.0:1788574546.907956:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574546.907961:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574546.907971:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569852416:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574546.907984:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574546.908001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569852672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574546.908014:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574546.908018:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574546.908021:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574546.908026:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11300 00000100:00080000:0.0:1788574546.908031:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574546.908036:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574546.908042:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574546.908055:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569852800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574546.908065:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574546.908071:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788574546.908072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574546.908074:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574546.908082:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11300 00000100:00080000:0.0:1788574546.908086:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574546.908091:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574546.908097:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574546.908122:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569852928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574546.908128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574546/real 1788574546] req@ffff920a48353800 x1875444569852416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574546.908159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569852416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574546.908173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574546.908178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574546.908183:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569852416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574546.908191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574546.908219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574546.908233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574546/real 1788574546] req@ffff920a48351c00 x1875444569852544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574546.908246:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569852544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574546.908253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574546.908257:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574546.908260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574546.908294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574546.908316:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574546.908329:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574546/real 1788574546] req@ffff920a48350a80 x1875444569852672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574546.908341:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569852672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574546.908381:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574546.908385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574546.908388:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574546.908394:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574546.908416:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574546.908427:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574546/real 1788574546] req@ffff920a48353480 x1875444569852800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574546.908439:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569852800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574546.908447:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574546.908450:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574546.908453:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574546.908460:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574546.908479:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574546.908571:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574546/real 1788574546] req@ffff920a48350700 x1875444569852928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574546.908616:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569852928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574571 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574546.908626:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574546.908630:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574546.908634:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569852928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574547.355877:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574547.931703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574548.379781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574548.955672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574549.403749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574549.979705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574550.427767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574551.003673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574551.451790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574552.027709:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574552.027725:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574552.027732:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574552.027743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574552.027748:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11305 00000100:00080000:3.0:1788574552.027763:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574552.027775:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574552.027827:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569853056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574552.027862:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574552.027866:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574552.027869:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574552.027875:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11305 00000100:00080000:3.0:1788574552.027881:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574552.027887:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574552.027892:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569853056:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574552.027902:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574552.027920:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569853184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574552.027933:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574552.027937:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574552.027940:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574552.027944:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11305 00000100:00080000:3.0:1788574552.027949:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574552.027955:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574552.027961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574552.027969:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574552.027977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569853312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574552.027989:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574552.027993:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574552.027995:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574552.028001:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11305 00000100:00080000:3.0:1788574552.028006:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574552.028008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff920a529b1f80 x1875444569853056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574552.028016:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574552.028025:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574552.028028:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569853056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574552.028041:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569853440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574552.028045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574552.028053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574552.028060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569853056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574552.028064:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574552.028069:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574552.028072:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574552.028077:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11305 00000100:00080000:3.0:1788574552.028082:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574552.028088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853184:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574552.028089:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574552.028099:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574552.028113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574552.028118:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569853568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574552.028125:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff920a529b0000 x1875444569853184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574552.028161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569853184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574552.028169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574552.028172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574552.028175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574552.028180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574552.028193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574552.028200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff920a529b0700 x1875444569853312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574552.028207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569853312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574552.028216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574552.028219:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574552.028223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574552.028229:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574552.028249:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574552.028259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff920a529b2300 x1875444569853440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574552.028271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569853440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574552.028279:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574552.028283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574552.028286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574552.028300:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574552.028321:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574552.028331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574552/real 1788574552] req@ffff920a529b0380 x1875444569853568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574552.028341:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569853568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574577 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574552.028349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574552.028352:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574552.028355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574552.475794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574553.051689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574553.499808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574554.075687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574554.523755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574555.099732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574555.547808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574556.123714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574556.571784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574557.147710:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574557.147719:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574557.147723:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574557.147733:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11310 00000100:00080000:3.0:1788574557.147743:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574557.147750:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574557.147756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788574557.147787:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569853696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574557.147810:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574557.147814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574557.147816:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574557.147820:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11310 00000100:00080000:3.0:1788574557.147824:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574557.147827:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574557.147837:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574557.147839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569853696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574557.147853:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569853824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574557.147864:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574557.147867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574557.147868:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574557.147872:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11310 00000100:00080000:3.0:1788574557.147876:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574557.147879:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574557.147884:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574557.147896:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569853952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574557.147908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574557.147911:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574557.147915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574557.147920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11310 00000100:00080000:3.0:1788574557.147925:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574557.147930:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574557.147938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574557.147940:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574557.147961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569854080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574557.147972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574557.147975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574557.147977:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574557.147981:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11310 00000100:00080000:3.0:1788574557.147986:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574557.147990:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574557.148021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff920a529b3b80 x1875444569853696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574557.148040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569853696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574557.148050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574557.148055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574557.148059:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569853696:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574557.148076:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574557.148093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574557.148094:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569854208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574557.148147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574557.148166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff920a529b0e00 x1875444569853824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574557.148181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569853824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574557.148189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574557.148192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574557.148195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574557.148200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574557.148220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574557.148230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff920a529b1180 x1875444569853952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574557.148247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569853952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574557.148258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574557.148263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574557.148266:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569853952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574557.148273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574557.148301:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574557.148319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff920a529b1500 x1875444569854080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574557.148331:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569854080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574557.148339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574557.148341:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574557.148345:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574557.148373:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574557.148405:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574557.148448:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574557/real 1788574557] req@ffff920a529b3b80 x1875444569854208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574557.148470:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569854208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574582 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574557.148478:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574557.148482:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574557.148486:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574557.595765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574558.171691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574558.619704:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574559.195714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574559.643771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574560.219657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574560.667849:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574561.243667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574561.691724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574562.267691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574562.267691:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.267704:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574562.267710:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.267724:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11315 00000100:00080000:3.0:1788574562.267737:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574562.267747:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574562.267797:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569854336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574562.267829:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.267833:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574562.267836:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.267841:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11315 00000100:00080000:3.0:1788574562.267845:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574562.267849:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574562.267860:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574562.267872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569854464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574562.267881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569854336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574562.267882:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.267891:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574562.267893:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.267897:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11315 00000100:00080000:3.0:1788574562.267899:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574562.267901:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574562.267904:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574562.267913:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569854592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574562.267924:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.267927:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574562.267928:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.267930:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11315 00000100:00080000:3.0:1788574562.267932:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574562.267935:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574562.267938:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574562.267944:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569854720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574562.267950:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574562.267953:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574562.267954:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574562.267956:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11315 00000100:00080000:3.0:1788574562.267958:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574562.267960:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574562.267963:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574562.267966:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574562.267974:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569854848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574562.268006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff920a529b0380 x1875444569854336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574562.268021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569854336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574562.268030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574562.268033:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574562.268036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569854336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574562.268058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574562.268073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574562.268081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff920a529b2300 x1875444569854464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574562.268089:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569854464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574562.268095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574562.268098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574562.268100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574562.268103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574562.268113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574562.268119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff920a529b0700 x1875444569854592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574562.268126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569854592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574562.268131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574562.268133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574562.268135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574562.268138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574562.268148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574562.268154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff920a529b0000 x1875444569854720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574562.268160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569854720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574562.268165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574562.268167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574562.268170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574562.268190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574562.268201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574562.268207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574562/real 1788574562] req@ffff920a529b1f80 x1875444569854848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574562.268214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569854848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574562.268220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574562.268221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574562.268223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569854848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574562.715771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574563.291663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574563.739743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574564.315704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574564.763765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574565.339672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574565.787788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574566.363752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574566.811755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574567.387690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574567.387690:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574567.387701:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574567.387707:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574567.387718:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11320 00000100:00080000:3.0:1788574567.387730:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574567.387738:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574567.387784:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569854976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574567.387812:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574567.387816:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574567.387818:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574567.387822:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11320 00000100:00080000:3.0:1788574567.387826:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574567.387831:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574567.387841:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574567.387852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569854976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574567.387859:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569855104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574567.387872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574567.387876:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574567.387878:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574567.387883:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11320 00000100:00080000:3.0:1788574567.387888:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574567.387892:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574567.387897:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574567.387910:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569855232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574567.387915:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574567.387925:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574567.387929:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574567.387932:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574567.387936:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11320 00000100:00080000:3.0:1788574567.387941:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574567.387946:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574567.387949:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff920a529b3100 x1875444569854976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574567.387953:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574567.387970:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569854976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574567.387973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569855360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574567.387991:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574567.387996:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574567.387999:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574567.388005:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569854976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574567.388005:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574567.388011:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574567.388020:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11320 00000100:00080000:3.0:1788574567.388024:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574567.388028:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574567.388034:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855104:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788574567.388034:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574567.388055:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569855488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574567.388061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574567.388073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff920a529b1c00 x1875444569855104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574567.388083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569855104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574567.388090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574567.388093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574567.388097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574567.388101:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574567.388113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574567.388119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff920a529b2a00 x1875444569855232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574567.388127:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569855232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574567.388132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574567.388134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574567.388136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574567.388139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574567.388150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574567.388155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff920a529b3480 x1875444569855360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574567.388164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569855360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574567.388187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574567.388191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574567.388195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574567.388213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574567.388231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574567.388239:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574567/real 1788574567] req@ffff920a529b2d80 x1875444569855488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574567.388247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569855488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574567.388253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574567.388255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574567.388257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574567.835782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574568.411690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574568.859748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574569.435685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574569.883752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574570.459694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574570.907769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574571.483683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574571.931757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574572.507706:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574572.507709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574572.507719:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574572.507724:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.507736:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11325 00000100:00080000:2.0:1788574572.507747:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.507755:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574572.507792:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569855616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.507819:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.507823:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574572.507825:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.507829:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11325 00000100:00080000:2.0:1788574572.507833:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.507837:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574572.507848:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574572.507861:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569855744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.507873:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574572.507879:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569855616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574572.507889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574572.507893:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.507898:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11325 00000100:00080000:2.0:1788574572.507901:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.507906:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574572.507911:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574572.507925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569855872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.507936:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.507940:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574572.507943:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.507947:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11325 00000100:00080000:2.0:1788574572.507950:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.507957:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574572.507958:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788574572.507964:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574572.507977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569856000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574572.507987:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574572.507990:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574572.507993:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574572.507999:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11325 00000100:00100000:0.0:1788574572.508000:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff920a60ac4a80 x1875444569855616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574572.508003:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574572.508010:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574572.508018:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574572.508032:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569856128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574572.508036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569855616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574572.508060:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574572.508066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574572.508071:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569855616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574572.508097:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574572.508127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574572.508144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff920a60ac4700 x1875444569855744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574572.508161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569855744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574572.508170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574572.508175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574572.508178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574572.508185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574572.508208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574572.508220:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff920a60ac4000 x1875444569855872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574572.508233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569855872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574572.508243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574572.508246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574572.508250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569855872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574572.508257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574572.508281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574572.508293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff920a60ac5c00 x1875444569856000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574572.508307:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569856000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574572.508349:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574572.508353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574572.508358:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574572.508365:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574572.508388:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574572.508400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574572/real 1788574572] req@ffff920a60ac4380 x1875444569856128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574572.508414:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569856128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574572.508423:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574572.508427:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574572.508430:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574572.955752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574573.531736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574573.979774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574574.555711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574575.003802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574575.579763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574576.027784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574576.603700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574577.051786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574577.627660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574577.627701:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574577.627708:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574577.627712:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574577.627721:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11331 00000100:00080000:3.0:1788574577.627731:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574577.627739:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574577.627778:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569856256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574577.627807:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574577.627811:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574577.627813:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574577.627818:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11331 00000100:00080000:3.0:1788574577.627822:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574577.627827:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574577.627839:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574577.627845:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569856256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574577.627859:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569856384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574577.627871:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574577.627875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574577.627878:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574577.627882:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11331 00000100:00080000:3.0:1788574577.627886:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574577.627890:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574577.627896:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574577.627910:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569856512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574577.627918:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574577.627928:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574577.627933:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574577.627936:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574577.627942:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11331 00000100:00080000:3.0:1788574577.627946:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574577.627951:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574577.627957:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574577.627961:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574577/real 1788574577] req@ffff920a529b2d80 x1875444569856256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574577.627973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569856640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574577.627987:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569856256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574577.627989:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574577.627998:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574577.628002:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574577.628012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574577.628012:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11331 00000100:00080000:3.0:1788574577.628018:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574577.628022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574577.628027:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574577.628030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569856256:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574577.628035:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574577.628051:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569856768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574577.628060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574577.628090:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574577.628104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574577/real 1788574577] req@ffff920a529b3480 x1875444569856384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574577.628117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569856384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574577.628125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574577.628129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574577.628132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574577.628138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574577.628157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574577.628168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574577/real 1788574577] req@ffff920a529b2a00 x1875444569856512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574577.628179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569856512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574577.628187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574577.628190:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574577.628193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574577.628199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574577.628217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574577.628226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574577/real 1788574577] req@ffff920a529b1c00 x1875444569856640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574577.628237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569856640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574577.628245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574577.628248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574577.628252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574577.628268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574577.628307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574577.628320:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574577/real 1788574577] req@ffff920a529b3100 x1875444569856768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574577.628341:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569856768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574577.628350:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574577.628354:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574577.628357:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569856768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574578.075756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574578.651685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574579.099815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574579.675737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574580.123862:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574580.699698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574581.147800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574581.723717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574582.171832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574582.747705:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574582.747713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574582.747714:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574582.747719:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574582.747730:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11336 00000100:00080000:3.0:1788574582.747740:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574582.747748:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574582.747783:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569856896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574582.747809:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574582.747813:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574582.747814:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574582.747817:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11336 00000100:00080000:3.0:1788574582.747820:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574582.747823:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574582.747831:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574582.747841:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569857024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574582.747849:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574582.747852:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574582.747853:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574582.747858:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11336 00000100:00100000:0.0:1788574582.747859:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569856896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574582.747861:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574582.747867:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574582.747871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574582.747882:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569857152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574582.747892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574582.747896:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574582.747898:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574582.747902:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11336 00000100:00080000:3.0:1788574582.747907:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574582.747912:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574582.747919:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574582.747933:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574582.747935:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569857280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574582.747945:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574582.747947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574582.747950:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574582.747953:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11336 00000100:00080000:3.0:1788574582.747956:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574582.747959:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574582.747964:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574582.747970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574582/real 1788574582] req@ffff920a529b1f80 x1875444569856896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574582.747975:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569857408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574582.748000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569856896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574582.748010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574582.748013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574582.748016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569856896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574582.748041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574582.748062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574582.748070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574582/real 1788574582] req@ffff920a529b0000 x1875444569857024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574582.748079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569857024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574582.748085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574582.748088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574582.748090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574582.748094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574582.748104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574582.748110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574582/real 1788574582] req@ffff920a529b0700 x1875444569857152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574582.748118:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569857152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574582.748123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574582.748125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574582.748127:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574582.748131:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574582.748141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574582.748147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574582/real 1788574582] req@ffff920a529b2300 x1875444569857280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574582.748160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569857280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574582.748165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574582.748167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574582.748168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574582.748172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574582.748182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574582.748187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574582/real 1788574582] req@ffff920a529b0380 x1875444569857408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574582.748200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569857408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574582.748206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574582.748208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574582.748210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574583.195713:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574583.771765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574584.219761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574584.795672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574585.243811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574585.819692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574586.267771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574586.843662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574587.291774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574587.867731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574587.867751:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574587.867760:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574587.867765:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574587.867776:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11341 00000100:00080000:1.0:1788574587.867787:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574587.867795:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574587.867839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569857536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574587.867863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574587.867866:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574587.867868:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574587.867871:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11341 00000100:00080000:1.0:1788574587.867874:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574587.867878:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574587.867890:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574587.867900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569857536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574587.867902:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569857664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574587.867921:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574587.867923:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574587.867925:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574587.867929:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11341 00000100:00080000:1.0:1788574587.867933:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574587.867936:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574587.867940:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574587.867948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569857792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574587.867955:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574587.867958:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574587.867960:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574587.867962:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11341 00000100:00080000:1.0:1788574587.867971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574587.867974:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574587.867977:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574587.867990:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569857920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574587.867992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574587.868002:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574587.868005:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574587.868008:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574587.868015:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11341 00000100:00080000:1.0:1788574587.868020:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574587.868023:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574587.868027:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574587.868035:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569858048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574587.868041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574587/real 1788574587] req@ffff920a52891f80 x1875444569857536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574587.868061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569857536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574587.868072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574587.868075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574587.868079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569857536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574587.868099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574587.868119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574587.868130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574587/real 1788574587] req@ffff920a52891500 x1875444569857664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574587.868143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569857664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574587.868153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574587.868168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574587.868172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574587.868180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574587.868199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574587.868208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574587/real 1788574587] req@ffff920a52892680 x1875444569857792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574587.868221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569857792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574587.868240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574587.868242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574587.868245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574587.868250:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574587.868266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574587.868274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574587/real 1788574587] req@ffff920a52892a00 x1875444569857920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574587.868281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569857920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574587.868286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574587.868289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574587.868291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569857920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574587.868294:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574587.868305:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574587.868310:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574587/real 1788574587] req@ffff920a52890a80 x1875444569858048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574587.868317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569858048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574587.868327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574587.868329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574587.868332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574588.315798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574588.891735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574589.339741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574589.915701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574590.363802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574590.939696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574591.387768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574591.963688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574592.411726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574592.987691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574592.987704:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574592.987710:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574592.987714:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574592.987723:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11346 00000100:00080000:3.0:1788574592.987734:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574592.987740:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574592.987775:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569858176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574592.987798:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574592.987801:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574592.987802:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574592.987805:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11346 00000100:00080000:3.0:1788574592.987809:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574592.987813:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574592.987822:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574592.987831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569858176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574592.987833:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569858304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574592.987844:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574592.987847:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574592.987848:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574592.987851:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11346 00000100:00080000:3.0:1788574592.987853:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574592.987856:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574592.987858:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574592.987866:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569858432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574592.987873:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574592.987875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574592.987876:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574592.987879:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11346 00000100:00080000:3.0:1788574592.987881:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574592.987883:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574592.987887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574592.987894:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569858560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574592.987900:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574592.987911:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574592.987914:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574592.987916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574592.987920:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11346 00000100:00080000:3.0:1788574592.987922:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574592.987924:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574592.987928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574592.987937:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574592/real 1788574592] req@ffff920a529b3b80 x1875444569858176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574592.987939:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569858688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574592.987978:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569858176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574592.987987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574592.987991:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574592.987995:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569858176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574592.988014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574592.988032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574592.988041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574592/real 1788574592] req@ffff920a529b1500 x1875444569858304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574592.988056:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569858304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574592.988063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574592.988065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574592.988069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574592.988077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574592.988103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574592.988118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574592/real 1788574592] req@ffff920a529b1180 x1875444569858432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574592.988145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569858432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574592.988156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574592.988161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574592.988166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574592.988174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574592.988217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574592.988232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574592/real 1788574592] req@ffff920a529b0e00 x1875444569858560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574592.988245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569858560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574592.988257:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574592.988261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574592.988264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574592.988272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574592.988295:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574592.988308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574592/real 1788574592] req@ffff920a529b0380 x1875444569858688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574592.988320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569858688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574617 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574592.988329:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574592.988333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574592.988337:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574593.435752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574594.011703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574594.459788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574595.035703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574595.483746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574596.059671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574596.507756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574597.083681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574597.531721:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574598.107686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574598.107692:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574598.107699:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574598.107703:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574598.107712:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11351 00000100:00080000:1.0:1788574598.107723:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574598.107732:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574598.107770:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569858816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574598.107802:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574598.107806:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574598.107809:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574598.107814:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11351 00000100:00080000:1.0:1788574598.107818:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574598.107824:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574598.107833:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569858816:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574598.107841:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574598.107854:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569858944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574598.107864:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574598.107867:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574598.107869:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574598.107873:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11351 00000100:00080000:1.0:1788574598.107883:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574598.107893:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788574598.107895:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788574598.107898:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574598.107909:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569859072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574598.107918:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574598.107922:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574598.107923:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574598.107927:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11351 00000100:00100000:0.0:1788574598.107928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff920a52890000 x1875444569858816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574598.107930:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574598.107934:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574598.107939:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574598.107947:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569858816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574598.107948:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569859200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574598.107958:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574598.107959:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574598.107962:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574598.107963:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788574598.107963:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574598.107967:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569858816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574598.107967:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11351 00000100:00080000:1.0:1788574598.107971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574598.107973:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574598.107992:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574598.108001:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569859328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574598.108005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574598.108031:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574598.108062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff920a52893b80 x1875444569858944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574598.108074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569858944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574598.108080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574598.108082:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574598.108085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569858944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574598.108089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574598.108104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574598.108111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff920a52890a80 x1875444569859072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574598.108118:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569859072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574598.108128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574598.108130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574598.108132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574598.108135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574598.108146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574598.108152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff920a52892a00 x1875444569859200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574598.108159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569859200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574598.108177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574598.108180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574598.108182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574598.108194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574598.108208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574598.108216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574598/real 1788574598] req@ffff920a52892680 x1875444569859328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574598.108223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569859328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574598.108228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574598.108230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574598.108233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574598.555869:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574599.131775:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574599.579774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574600.155768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574600.603740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574601.179731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574601.627774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574602.203692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574602.651741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574603.227752:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574603.227765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574603.227766:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574603.227772:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574603.227786:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11356 00000100:00080000:2.0:1788574603.227804:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574603.227813:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574603.227867:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569859456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574603.227895:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574603.227898:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574603.227900:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574603.227903:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11356 00000100:00080000:2.0:1788574603.227906:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574603.227908:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574603.227918:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574603.227937:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569859584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574603.227952:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574603.227955:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569859456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574603.227957:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574603.227959:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574603.227970:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11356 00000100:00080000:2.0:1788574603.227979:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574603.227982:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574603.227986:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574603.227995:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569859712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574603.228004:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574603.228007:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574603.228008:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574603.228011:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11356 00000100:00080000:2.0:1788574603.228012:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574603.228015:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574603.228018:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574603.228024:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569859840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574603.228031:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574603.228034:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574603.228035:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574603.228037:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11356 00000100:00080000:2.0:1788574603.228046:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574603.228049:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574603.228054:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574603.228062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574603.228067:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569859968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574603.228122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff920a60ac4e00 x1875444569859456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.228164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569859456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.228178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574603.228183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574603.228188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569859456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574603.228215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574603.228246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574603.228262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff920a60ac7800 x1875444569859584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.228277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569859584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.228287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574603.228291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574603.228294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574603.228302:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574603.228327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574603.228341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff920a60ac4380 x1875444569859712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.228355:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569859712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.228364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574603.228368:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574603.228372:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574603.228380:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574603.228404:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574603.228416:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff920a60ac5c00 x1875444569859840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.228427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569859840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.228436:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574603.228439:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574603.228442:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574603.228449:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574603.228471:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574603.228497:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574603/real 1788574603] req@ffff920a60ac4000 x1875444569859968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574603.228519:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569859968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574603.228528:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574603.228532:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574603.228536:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569859968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574603.675789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574604.251705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574604.699737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574605.275740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574605.723780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574606.299684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574606.747741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574607.323704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574607.771820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574608.347713:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574608.347714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574608.347730:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574608.347734:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.347744:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11361 00000100:00080000:2.0:1788574608.347755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.347764:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574608.347799:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569860096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.347828:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.347831:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574608.347833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.347836:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11361 00000100:00080000:2.0:1788574608.347838:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.347840:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574608.347848:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574608.347859:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569860224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.347867:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.347871:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574608.347874:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.347880:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11361 00000100:00100000:0.0:1788574608.347888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569860096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574608.347889:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.347906:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574608.347914:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574608.347952:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569860352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.347966:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.347969:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574608.347971:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.347974:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11361 00000100:00080000:2.0:1788574608.347980:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.347984:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574608.347991:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574608.348000:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569860480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574608.348007:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574608.348010:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574608.348011:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574608.348013:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11361 00000100:00080000:2.0:1788574608.348016:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574608.348019:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574608.348023:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574608.348043:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569860608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574608.348096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574608.348198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574608.348222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574608.348245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574608.795751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574609.371715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574609.819754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574610.395739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574610.843761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574611.419705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788574611.419870:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788574611.419919:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788574611.419934:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788574611.419971:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788574611.419983:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788574611.419998:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788574611.420019:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788574611.420021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574611] req@ffff920a60ac5500 x1875444569860352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574611.420055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569860352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574611.420067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574611.420072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574611.420076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574611.420100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574611] req@ffff920a60ac6300 x1875444569860480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574611.420112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569860480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574611.420118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574611.420121:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574611.420123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574611.420128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574611] req@ffff920a60ac7100 x1875444569860608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574611.420140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569860608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574611.420149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574611.420153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574611.420156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860608:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574611.420163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574611] req@ffff920a60ac4700 x1875444569860096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574611.420176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569860096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574611.420185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574611.420189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574611.420193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569860096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574611.420201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574608/real 1788574611] req@ffff920a60ac4a80 x1875444569860224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574611.420215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569860224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574611.420222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574611.420226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574611.420229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860224:192.168.202.157@tcp:38:kworker.0 00010000:00010000:0.0:1788574611.867764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574612.443717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574612.891772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574613.467706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574613.467709:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574613.467723:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574613.467730:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574613.467743:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11366 00000100:00080000:2.0:1788574613.467755:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574613.467764:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574613.467808:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569860736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574613.467840:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574613.467844:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574613.467847:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574613.467852:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11366 00000100:00080000:2.0:1788574613.467856:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574613.467860:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574613.467875:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574613.467891:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569860864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574613.467906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569860736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574613.467906:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574613.467917:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574613.467921:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574613.467927:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11366 00000100:00080000:2.0:1788574613.467931:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574613.467935:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574613.467943:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574613.467960:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569860992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574613.467972:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574613.467976:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574613.467979:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574613.467984:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11366 00000100:00080000:2.0:1788574613.467988:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574613.467993:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574613.468000:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574613.468012:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569861120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574613.468018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574613.468031:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574613.468036:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574613.468039:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574613.468045:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11366 00000100:00080000:2.0:1788574613.468051:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574613.468056:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574613.468062:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574613.468099:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569861248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574613.468123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff920a60ac5180 x1875444569860736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574613.468151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569860736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574613.468166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574613.468175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574613.468180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569860736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574613.468215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574613.468265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574613.468281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff920a60ac6a00 x1875444569860864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574613.468302:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569860864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574613.468318:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574613.468321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574613.468325:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574613.468333:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574613.468357:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574613.468371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff920a60ac4a80 x1875444569860992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574613.468385:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569860992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574613.468400:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574613.468404:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574613.468408:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569860992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574613.468416:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574613.468442:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574613.468456:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff920a60ac4700 x1875444569861120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574613.468471:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569861120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574613.468488:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574613.468492:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574613.468496:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574613.468503:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574613.468529:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574613.468542:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574613/real 1788574613] req@ffff920a60ac7100 x1875444569861248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574613.468556:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569861248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574613.468565:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574613.468568:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574613.468572:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574613.915820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574614.491700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574614.939796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574615.515675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574615.963771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574616.539690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574616.987799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574617.563676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574618.011768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574618.587745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574618.587745:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574618.587762:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574618.587768:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574618.587781:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11371 00000100:00080000:2.0:1788574618.587794:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574618.587805:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574618.587857:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569861376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574618.587886:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574618.587889:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574618.587891:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574618.587894:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11371 00000100:00080000:2.0:1788574618.587897:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574618.587901:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574618.587913:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574618.587925:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569861504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574618.587934:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574618.587937:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574618.587939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574618.587946:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11371 00000100:00100000:0.0:1788574618.587948:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569861376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574618.587951:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574618.587957:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574618.587966:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574618.587975:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569861632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574618.587987:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574618.587990:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574618.587991:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574618.587994:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11371 00000100:00080000:2.0:1788574618.587996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574618.587999:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574618.588001:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574618.588009:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569861760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574618.588014:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574618.588016:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574618.588017:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574618.588020:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11371 00000100:00080000:2.0:1788574618.588023:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574618.588026:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574618.588029:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574618.588035:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574618.588037:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569861888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574618.588102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff920a60ac6300 x1875444569861376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574618.588124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569861376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574618.588144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574618.588148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574618.588153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569861376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574618.588180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574618.588207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574618.588221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff920a60ac5500 x1875444569861504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574618.588234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569861504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574618.588241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574618.588245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574618.588248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574618.588255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574618.588275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574618.588286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff920a60ac4000 x1875444569861632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574618.588296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569861632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574618.588304:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574618.588307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574618.588311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574618.588318:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574618.588337:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574618.588346:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff920a60ac5c00 x1875444569861760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574618.588370:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569861760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574618.588378:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574618.588381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574618.588385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574618.588391:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574618.588412:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574618.588423:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574618/real 1788574618] req@ffff920a60ac4380 x1875444569861888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574618.588435:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569861888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574618.588443:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574618.588447:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574618.588450:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569861888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574619.035757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574619.611673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574620.059744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574620.635695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574621.083784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574621.659716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574622.107729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574622.683686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574623.131776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574623.707661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574623.707721:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574623.707729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574623.707733:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574623.707743:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11377 00000100:00080000:2.0:1788574623.707755:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574623.707764:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574623.707805:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569862016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574623.707836:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574623.707841:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574623.707844:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574623.707849:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11377 00000100:00080000:2.0:1788574623.707854:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574623.707859:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574623.707871:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574623.707885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569862016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574623.707888:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569862144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574623.707908:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574623.707912:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574623.707915:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574623.707920:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11377 00000100:00080000:2.0:1788574623.707925:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574623.707930:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574623.707937:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574623.707951:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569862272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574623.707962:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574623.707967:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574623.707973:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574623.707976:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574623.707982:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11377 00000100:00080000:2.0:1788574623.707987:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574623.708005:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574623/real 1788574623] req@ffff920a60ac7800 x1875444569862016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788574623.708014:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574623.708034:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569862400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574623.708039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569862016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574623.708058:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574623.708061:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574623.708065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574623.708070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569862016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574623.708075:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574623.708079:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574623.708085:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11377 00000100:00080000:2.0:1788574623.708090:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574623.708094:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862144:192.168.202.157@tcp:38:kworker.0 00000008:00100000:2.0:1788574623.708101:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574623.708121:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569862528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574623.708134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574623/real 1788574623] req@ffff920a60ac4e00 x1875444569862144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574623.708158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569862144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574623.708165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574623.708177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574623.708183:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574623.708190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574623.708211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574623/real 1788574623] req@ffff920a60ac4380 x1875444569862272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574623.708244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569862272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574623.708250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574623.708252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574623.708254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574623.708259:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574623.708274:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574623/real 1788574623] req@ffff920a60ac5c00 x1875444569862400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574623.708289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569862400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574623.708294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574623.708295:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574623.708297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574623.708308:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574623.708319:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574623.708325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574623/real 1788574623] req@ffff920a60ac4000 x1875444569862528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574623.708332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569862528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574623.708342:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574623.708344:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574623.708346:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574624.155762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574624.731732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574625.179778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574625.755691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574626.203751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574626.779684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574627.227792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574627.803727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574628.251811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574628.827675:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574628.827687:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574628.827693:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574628.827712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574628.827714:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11382 00000100:00080000:0.0:1788574628.827738:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574628.827751:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574628.827796:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569862656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574628.827824:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574628.827828:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574628.827830:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574628.827835:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11382 00000100:00080000:0.0:1788574628.827840:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574628.827843:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574628.827855:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574628.827868:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569862784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574628.827868:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569862656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574628.827892:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574628.827895:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574628.827897:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574628.827901:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11382 00000100:00080000:0.0:1788574628.827906:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574628.827912:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574628.827915:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574628.827926:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569862912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574628.827933:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574628.827937:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574628.827939:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574628.827955:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11382 00000100:00080000:0.0:1788574628.827958:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574628.827965:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788574628.827967:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788574628.827971:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574628.827985:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569863040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574628.827992:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574628.827995:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574628.827997:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574628.828000:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11382 00000100:00080000:0.0:1788574628.828004:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574628.828008:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574628/real 1788574628] req@ffff920a48350380 x1875444569862656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574628.828011:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574628.828018:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574628.828028:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569863168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574628.828033:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569862656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574628.828055:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574628.828059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574628.828063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569862656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574628.828088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574628.828107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574628.828117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574628/real 1788574628] req@ffff920a48353b80 x1875444569862784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574628.828125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569862784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574628.828131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574628.828133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574628.828135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574628.828139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574628.828150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574628.828156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574628/real 1788574628] req@ffff920a48351500 x1875444569862912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574628.828164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569862912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574628.828169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574628.828171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574628.828173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569862912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574628.828177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574628.828188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574628.828193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574628/real 1788574628] req@ffff920a48350700 x1875444569863040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574628.828201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569863040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574628.828206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574628.828208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574628.828210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574628.828213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574628.828223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574628.828228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574628/real 1788574628] req@ffff920a48353480 x1875444569863168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574628.828236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569863168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574628.828243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574628.828247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574628.828250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574629.275787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574629.851696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574630.299778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574630.875705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574631.323763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574631.899681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574632.347742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574632.923746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574633.371832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574633.947725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574633.947730:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574633.947740:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574633.947744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574633.947754:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11387 00000100:00080000:3.0:1788574633.947768:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574633.947778:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574633.947828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569863296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574633.947870:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574633.947873:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574633.947875:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574633.947879:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11387 00000100:00080000:3.0:1788574633.947882:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574633.947885:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574633.947909:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574633.947923:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569863424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574633.947930:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569863296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574633.947942:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574633.947946:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574633.947948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574633.947954:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11387 00000100:00080000:3.0:1788574633.947959:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574633.947963:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574633.947970:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574633.947983:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569863552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574633.947993:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574633.947998:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574633.948001:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574633.948005:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11387 00000100:00080000:3.0:1788574633.948009:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574633.948015:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574633.948016:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574633.948035:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574633.948048:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569863680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574633.948058:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574633.948062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574633/real 1788574633] req@ffff920a529b2300 x1875444569863296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574633.948064:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574633.948069:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574633.948080:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11387 00000100:00080000:3.0:1788574633.948085:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574633.948088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569863296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574633.948094:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788574633.948102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788574633.948103:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574633.948107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574633.948113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569863296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574633.948122:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569863808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574633.948139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574633.948157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574633.948166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574633/real 1788574633] req@ffff920a529b0700 x1875444569863424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574633.948179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569863424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574633.948188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574633.948192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574633.948196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574633.948201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574633.948223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574633.948236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574633/real 1788574633] req@ffff920a529b0000 x1875444569863552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574633.948249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569863552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574633.948265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574633.948269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574633.948273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574633.948279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574633.948300:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574633.948313:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574633/real 1788574633] req@ffff920a529b1f80 x1875444569863680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574633.948326:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569863680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574633.948334:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574633.948337:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574633.948341:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574633.948348:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574633.948368:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574633.948379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574633/real 1788574633] req@ffff920a529b3100 x1875444569863808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574633.948388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569863808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574658 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574633.948393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574633.948395:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574633.948397:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569863808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574634.395773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574634.971712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574635.419811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574635.995702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574636.443744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574637.019711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574637.467773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574638.043710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574638.491761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574639.067701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574639.067745:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574639.067754:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574639.067759:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574639.067774:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11392 00000100:00080000:1.0:1788574639.067787:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574639.067795:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574639.067860:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569863936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574639.067887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574639.067890:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574639.067892:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574639.067895:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11392 00000100:00080000:1.0:1788574639.067898:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574639.067901:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574639.067912:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574639.067926:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569864064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574639.067941:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569863936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574639.067946:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574639.067955:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574639.067957:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574639.067964:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11392 00000100:00080000:1.0:1788574639.067970:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574639.067977:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574639.067984:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574639.067997:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569864192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574639.068004:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574639.068007:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574639.068009:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574639.068012:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11392 00000100:00080000:1.0:1788574639.068016:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574639.068023:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788574639.068023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788574639.068035:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574639.068046:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569864320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574639.068054:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574639.068058:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574639.068060:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574639.068065:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11392 00000100:00080000:1.0:1788574639.068069:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574639.068072:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574639.068077:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574639.068084:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff920a52891500 x1875444569863936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574639.068091:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569864448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574639.068163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569863936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574639.068178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574639.068183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574639.068188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569863936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574639.068216:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574639.068245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574639.068257:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff920a52891f80 x1875444569864064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574639.068267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569864064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574639.068274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574639.068276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574639.068279:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574639.068283:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574639.068298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574639.068305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff920a52891880 x1875444569864192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574639.068314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569864192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574639.068324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574639.068327:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574639.068331:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574639.068337:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574639.068360:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574639.068371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff920a52893480 x1875444569864320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574639.068390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569864320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574639.068398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574639.068401:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574639.068403:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574639.068408:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574639.068423:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574639.068430:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574639/real 1788574639] req@ffff920a52890380 x1875444569864448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574639.068441:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569864448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574639.068450:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574639.068453:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574639.068457:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574639.515734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574640.091697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574640.539737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574641.115691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574641.563758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574642.139706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574642.587744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574643.163815:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574643.611807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574644.187712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574644.187714:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574644.187724:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574644.187730:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574644.187741:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11397 00000100:00080000:3.0:1788574644.187752:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574644.187760:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574644.187822:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569864576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574644.187846:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574644.187849:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574644.187850:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574644.187853:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11397 00000100:00080000:3.0:1788574644.187856:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574644.187858:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574644.187868:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574644.187879:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569864704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574644.187885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569864576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574644.187889:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574644.187894:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574644.187895:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574644.187899:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11397 00000100:00080000:3.0:1788574644.187901:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574644.187903:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574644.187907:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574644.187917:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569864832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574644.187928:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574644.187931:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574644.187932:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574644.187935:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11397 00000100:00080000:3.0:1788574644.187952:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574644.187955:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574644.187958:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574644.187964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574644.187968:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569864960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574644.187976:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574644.187979:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574644.187981:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574644.187984:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11397 00000100:00080000:3.0:1788574644.187989:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574644.187992:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574644.187995:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574644.188009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff920a529b1c00 x1875444569864576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574644.188010:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569865088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574644.188039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569864576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574644.188050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574644.188054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574644.188057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569864576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574644.188079:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574644.188095:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574644.188104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff920a529b2a00 x1875444569864704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574644.188112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569864704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574644.188118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574644.188120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574644.188122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574644.188127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574644.188139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574644.188145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff920a529b3480 x1875444569864832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574644.188153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569864832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574644.188158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574644.188160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574644.188162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574644.188165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574644.188177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574644.188183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff920a529b2d80 x1875444569864960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574644.188190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569864960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574644.188195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574644.188197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574644.188199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569864960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574644.188202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574644.188212:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574644.188217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574644/real 1788574644] req@ffff920a529b3100 x1875444569865088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574644.188225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569865088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574644.188230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574644.188232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574644.188234:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574644.635848:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574645.211721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574645.659785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574646.235705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574646.683754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574647.259817:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574647.707774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574648.283714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574648.731770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574649.307697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574649.307725:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.307747:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574649.307752:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.307760:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11402 00000100:00080000:1.0:1788574649.307770:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.307775:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574649.307811:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569865216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.307841:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.307844:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574649.307845:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.307849:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11402 00000100:00080000:1.0:1788574649.307852:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.307855:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574649.307864:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574649.307875:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569865344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.307883:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.307887:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574649.307888:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574649.307888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569865216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574649.307898:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11402 00000100:00080000:1.0:1788574649.307903:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.307908:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574649.307914:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574649.307928:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569865472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.307939:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.307943:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574649.307945:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.307948:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11402 00000100:00080000:1.0:1788574649.307952:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574649.307956:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574649.307962:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574649.307966:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574649.307978:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569865600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574649.307990:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574649.307993:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574649.307995:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574649.308000:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11402 00000100:00080000:1.0:1788574649.308005:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574649.308007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff920a52890e00 x1875444569865216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574649.308011:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574649.308018:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574649.308031:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569865216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574649.308035:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569865728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574649.308047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574649.308054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574649.308058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569865216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574649.308080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574649.308098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574649.308107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff920a52893800 x1875444569865344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574649.308120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569865344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574649.308126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574649.308128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574649.308132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574649.308138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574649.308158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574649.308170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff920a52892d80 x1875444569865472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574649.308181:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569865472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574649.308189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574649.308193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574649.308196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574649.308202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574649.308223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574649.308234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff920a52892300 x1875444569865600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574649.308245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569865600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574649.308258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574649.308262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574649.308265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574649.308270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574649.308291:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574649.308302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574649/real 1788574649] req@ffff920a52891180 x1875444569865728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574649.308313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569865728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574649.308321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574649.308324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574649.308328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574649.755795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574650.331810:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574650.779775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574651.355741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574651.803768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574652.379736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574652.827773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574653.403721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574653.851761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574654.427747:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574654.427748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574654.427763:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574654.427768:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.427779:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11407 00000100:00080000:1.0:1788574654.427790:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.427797:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574654.427835:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569865856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.427856:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.427859:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574654.427860:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.427864:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11407 00000100:00080000:1.0:1788574654.427866:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.427870:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574654.427882:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574654.427896:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569865984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574654.427905:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569865856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574654.427919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.427925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574654.427927:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.427933:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11407 00000100:00080000:1.0:1788574654.427939:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.427943:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574654.427948:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574654.427962:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569866112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.427970:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.427972:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574654.427973:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.427976:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11407 00000100:00080000:1.0:1788574654.427980:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.427982:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574654.427987:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574654.427993:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574654.428004:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569866240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574654.428013:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574654.428016:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574654.428018:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574654.428022:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11407 00000100:00080000:1.0:1788574654.428028:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574654.428032:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574654.428037:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574654.428042:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff920a52893100 x1875444569865856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574654.428048:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569866368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574654.428110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569865856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574654.428121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574654.428125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574654.428130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569865856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574654.428157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574654.428180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574654.428193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff920a52890700 x1875444569865984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574654.428207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569865984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574654.428216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574654.428220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574654.428224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569865984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574654.428231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574654.428254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574654.428265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff920a52891180 x1875444569866112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574654.428273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569866112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574654.428280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574654.428282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574654.428285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574654.428289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574654.428305:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574654.428311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff920a52892300 x1875444569866240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574654.428320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569866240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574654.428325:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574654.428327:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574654.428329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574654.428332:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574654.428343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574654.428349:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574654/real 1788574654] req@ffff920a52892d80 x1875444569866368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574654.428357:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569866368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574654.428362:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574654.428364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574654.428365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574654.875792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574655.451772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574655.899739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574656.475728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574656.923808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574657.499732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574657.947783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574658.523701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574658.971800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574659.547735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574659.547738:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574659.547750:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574659.547755:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574659.547768:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11412 00000100:00080000:2.0:1788574659.547781:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574659.547790:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574659.547852:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569866496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574659.547878:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574659.547881:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574659.547883:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574659.547886:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11412 00000100:00080000:2.0:1788574659.547907:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574659.547911:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574659.547922:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574659.547930:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569866496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574659.547936:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569866624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574659.547945:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574659.547947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574659.547948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574659.547950:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11412 00000100:00080000:2.0:1788574659.547953:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574659.547955:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574659.547958:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574659.547965:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569866752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574659.547972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574659.547975:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574659.547976:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574659.547980:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11412 00000100:00080000:2.0:1788574659.547983:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574659.547986:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574659.547989:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574659.547996:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569866880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574659.548003:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574659.548005:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574659.548006:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574659.548011:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11412 00000800:00000100:0.0:1788574659.548012:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574659.548015:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574659.548019:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574659.548022:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574659.548029:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569867008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574659.548061:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574659] req@ffff920a60ac5500 x1875444569866496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574659.548082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569866496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574659.548090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574659.548094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574659.548098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569866496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574659.548119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574659.548136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574659.548144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574659] req@ffff920a60ac6300 x1875444569866624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574659.548152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569866624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574659.548158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574659.548160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574659.548162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574659.548166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574659.548176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574659.548182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574659] req@ffff920a60ac7100 x1875444569866752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574659.548189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569866752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574659.548198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574659.548200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574659.548203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574659.548206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574659.548217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574659.548223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574659] req@ffff920a60ac4700 x1875444569866880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574659.548230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569866880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574659.548235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574659.548237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574659.548239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569866880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574659.548242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574659.548254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574659.548259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574659/real 1788574659] req@ffff920a60ac4a80 x1875444569867008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574659.548266:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569867008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574659.548271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574659.548272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574659.548275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574659.995768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574660.571687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574661.019762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574661.595723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574662.043780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574662.619717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574663.067775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574663.643731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574664.091810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574664.667713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574664.667742:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574664.667749:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574664.667753:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574664.667762:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11418 00000100:00080000:0.0:1788574664.667772:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574664.667778:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574664.667812:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569867136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574664.667832:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574664.667836:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574664.667838:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574664.667842:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11418 00000100:00080000:0.0:1788574664.667846:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574664.667851:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574664.667863:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574664.667875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569867136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574664.667892:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569867264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574664.667909:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574664.667914:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574664.667916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574664.667921:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11418 00000100:00080000:0.0:1788574664.667927:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574664.667933:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574664.667941:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574664.667957:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569867392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574664.667970:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574664.667979:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574664.667980:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574664.667985:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574664.667992:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11418 00000100:00080000:0.0:1788574664.667996:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574664.668002:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574664.668008:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574664.668026:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569867520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574664.668032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574664/real 1788574664] req@ffff920a48350a80 x1875444569867136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574664.668061:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574664.668068:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574664.668071:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574664.668076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569867136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574664.668080:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11418 00000100:00080000:0.0:1788574664.668093:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574664.668104:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788574664.668104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788574664.668116:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574664.668117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574664.668132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569867136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574664.668142:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569867648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574664.668172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574664.668201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574664.668212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574664/real 1788574664] req@ffff920a48351c00 x1875444569867264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574664.668222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569867264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574664.668229:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574664.668233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574664.668236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574664.668241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574664.668261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574664.668271:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574664/real 1788574664] req@ffff920a48353800 x1875444569867392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574664.668283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569867392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574664.668289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574664.668292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574664.668294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574664.668301:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574664.668318:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574664.668334:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574664/real 1788574664] req@ffff920a48353480 x1875444569867520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574664.668350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569867520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574664.668357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574664.668360:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574664.668363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574664.668369:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574664.668383:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574664.668390:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574664/real 1788574664] req@ffff920a48350700 x1875444569867648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574664.668397:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569867648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574664.668402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574664.668404:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574664.668406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574665.115793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574665.691730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574666.139790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574666.715713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574667.163788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574667.739709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574668.187796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574668.763687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574669.211794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574669.787719:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574669.787732:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1788574669.787737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574669.787738:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574669.787754:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11423 00000100:00080000:3.0:1788574669.787767:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574669.787776:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574669.787821:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569867776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574669.787850:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574669.787854:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574669.787855:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574669.787858:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11423 00000100:00080000:3.0:1788574669.787861:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574669.787864:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574669.787875:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574669.787886:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569867904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574669.787895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569867776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574669.787919:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574669.787922:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574669.787925:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574669.787931:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11423 00000100:00080000:3.0:1788574669.787934:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574669.787937:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574669.787942:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574669.787952:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569868032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574669.787959:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574669.787962:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574669.787963:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574669.787968:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11423 00000100:00080000:3.0:1788574669.787971:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574669.787974:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788574669.787980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574669.787980:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574669.787995:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569868160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574669.788005:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574669.788008:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574669.788011:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574669.788017:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11423 00000100:00100000:2.0:1788574669.788029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574669/real 1788574669] req@ffff920a529b1f80 x1875444569867776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574669.788030:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574669.788044:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574669.788061:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574669.788069:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569867776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574669.788075:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569868288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574669.788120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574669.788128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574669.788132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569867776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574669.788156:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574669.788182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574669.788191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574669/real 1788574669] req@ffff920a529b0000 x1875444569867904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574669.788199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569867904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574669.788205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574669.788207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574669.788210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569867904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574669.788214:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574669.788225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574669.788231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574669/real 1788574669] req@ffff920a529b0700 x1875444569868032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574669.788239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569868032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574669.788249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574669.788251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574669.788253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574669.788257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574669.788268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574669.788273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574669/real 1788574669] req@ffff920a529b2300 x1875444569868160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574669.788282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569868160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574669.788290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574669.788294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574669.788297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574669.788303:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574669.788325:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574669.788337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574669/real 1788574669] req@ffff920a529b0380 x1875444569868288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574669.788346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569868288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574669.788351:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574669.788353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574669.788356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574670.235789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574670.811678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574671.259760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574671.835742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574672.283784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574672.859704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574673.307743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574673.883731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574674.331770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574674.907694:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574674.907704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574674.907709:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574674.907717:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574674.907731:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11428 00000100:00080000:2.0:1788574674.907745:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574674.907755:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574674.907808:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569868416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574674.907849:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574674.907861:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574674.907864:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574674.907869:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11428 00000100:00080000:2.0:1788574674.907875:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574674.907881:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574674.907895:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574674.907901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569868416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574674.907914:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569868544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574674.907927:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574674.907931:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574674.907934:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574674.907939:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11428 00000100:00080000:2.0:1788574674.907944:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574674.907948:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574674.907954:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574674.907978:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569868672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574674.907989:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574674.907993:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574674.907995:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574674.908000:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11428 00000100:00080000:2.0:1788574674.908004:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574674.908009:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574674.908015:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574674.908028:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569868800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574674.908038:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574674.908042:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574674.908045:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574674.908050:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788574674.908053:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11428 00000100:00080000:2.0:1788574674.908062:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574674.908067:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574674.908074:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574674.908090:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569868928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574674.908099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574674.908136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574674.908163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574675.355756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574675.931798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574676.379752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574676.955748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574677.403790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574677.979838:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788574677.979848:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788574677.979885:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788574677.979896:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574677.979927:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574677.979933:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574677.979946:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788574677.979955:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788574677.980006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574674/real 1788574677] req@ffff920a60ac7100 x1875444569868928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574677.980043:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569868928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574677.980057:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574677.980063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574677.980068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868928:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574677.980078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574674/real 1788574677] req@ffff920a60ac5180 x1875444569868544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574677.980093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569868544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574677.980102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574677.980106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574677.980110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574677.980118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574674/real 1788574677] req@ffff920a60ac4a80 x1875444569868672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574677.980130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569868672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574677.980137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574677.980141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574677.980144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574677.980151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574674/real 1788574677] req@ffff920a60ac4700 x1875444569868800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574677.980163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569868800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574677.980171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574677.980175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574677.980178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569868800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574677.980187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574674/real 1788574677] req@ffff920a60ac6a00 x1875444569868416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574677.980200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569868416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574677.980208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574677.980224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574677.980228:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569868416:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788574678.427741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574679.003756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574679.451756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574680.027665:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.027673:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574680.027676:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.027684:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11433 00000100:00080000:0.0:1788574680.027693:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.027699:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574680.027701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788574680.027727:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569869056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.027747:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.027750:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574680.027751:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.027754:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11433 00000100:00080000:0.0:1788574680.027756:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.027758:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574680.027765:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574680.027776:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569869184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.027786:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.027789:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574680.027791:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574680.027796:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569869056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574680.027798:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11433 00000100:00080000:0.0:1788574680.027805:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.027810:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574680.027815:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574680.027834:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569869312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.027848:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.027851:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574680.027852:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.027855:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11433 00000100:00080000:0.0:1788574680.027858:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.027861:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574680.027866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574680.027867:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574680.027880:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569869440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574680.027889:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574680.027893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574680.027894:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574680.027898:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11433 00000100:00100000:2.0:1788574680.027903:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff920a48351500 x1875444569869056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574680.027904:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574680.027911:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574680.027919:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574680.027927:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569869056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574680.027939:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569869568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574680.027939:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574680.027948:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574680.027955:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569869056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574680.027982:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574680.028004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574680.028015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff920a48353b80 x1875444569869184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574680.028026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569869184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574680.028034:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574680.028037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574680.028040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574680.028045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574680.028061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574680.028070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff920a48350380 x1875444569869312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574680.028081:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569869312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574680.028088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574680.028091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574680.028094:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574680.028100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574680.028117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574680.028126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff920a48350700 x1875444569869440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574680.028136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569869440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574680.028143:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574680.028146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574680.028150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574680.028155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574680.028172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574680.028180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574680/real 1788574680] req@ffff920a48353480 x1875444569869568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574680.028190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569869568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574680.028197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574680.028200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574680.028203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574680.475790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574681.051827:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574681.499789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574682.075693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574682.523769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574683.099686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574683.547751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574684.123676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574684.571753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574685.147691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574685.147724:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.147729:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574685.147732:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.147741:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11438 00000100:00080000:2.0:1788574685.147749:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.147755:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574685.147784:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569869696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.147811:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.147814:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574685.147815:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.147818:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11438 00000100:00080000:2.0:1788574685.147820:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.147822:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574685.147830:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574685.147839:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569869824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.147846:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.147848:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574685.147849:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574685.147853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569869696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574685.147853:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11438 00000100:00080000:2.0:1788574685.147858:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.147863:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574685.147866:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574685.147872:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569869952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.147881:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.147883:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574685.147884:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.147887:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11438 00000100:00080000:2.0:1788574685.147889:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.147890:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574685.147893:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574685.147900:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569870080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574685.147904:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574685.147907:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574685.147908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574685.147911:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11438 00000100:00080000:2.0:1788574685.147913:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574685.147914:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574685.147917:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574685.147922:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574685.147924:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569870208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574685.147960:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff920a60ac6a00 x1875444569869696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.147977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569869696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.147987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.147991:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574685.147995:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569869696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574685.148019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574685.148042:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574685.148054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff920a60ac4700 x1875444569869824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.148065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569869824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.148074:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.148077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574685.148081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574685.148086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574685.148102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574685.148111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff920a60ac4a80 x1875444569869952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.148122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569869952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.148135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.148138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574685.148141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569869952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574685.148147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574685.148163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574685.148171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff920a60ac5180 x1875444569870080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.148182:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569870080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.148190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.148192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574685.148195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574685.148201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574685.148227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574685.148236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574685/real 1788574685] req@ffff920a60ac7100 x1875444569870208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574685.148246:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569870208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574685.148253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574685.148256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574685.148259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574685.595751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574686.171703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574686.619733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574687.195714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574687.643771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574688.219674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574688.667736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574689.243686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574689.691761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574690.267692:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574690.267701:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574690.267706:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574690.267714:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11443 00000100:00080000:1.0:1788574690.267724:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574690.267731:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574690.267756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788574690.267763:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569870336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574690.267790:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574690.267793:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574690.267794:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574690.267798:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11443 00000100:00080000:1.0:1788574690.267800:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574690.267802:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574690.267810:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574690.267819:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569870464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574690.267826:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574690.267844:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574690.267847:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574690.267854:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11443 00000100:00100000:0.0:1788574690.267855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569870336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574690.267857:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574690.267865:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574690.267869:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574690.267880:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569870592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574690.267893:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574690.267896:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574690.267897:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574690.267900:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11443 00000100:00080000:1.0:1788574690.267903:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574690.267905:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574690.267909:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574690.267916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574690.267919:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569870720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574690.267927:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574690.267930:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574690.267931:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574690.267934:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11443 00000100:00080000:1.0:1788574690.267937:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574690.267939:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574690.267943:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574690.267951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff920a52893800 x1875444569870336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574690.267953:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569870848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574690.267981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569870336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.267990:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.267993:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574690.267995:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569870336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574690.268001:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574690.268018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574690.268027:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff920a52890e00 x1875444569870464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.268035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569870464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.268040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.268042:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574690.268044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574690.268064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574690.268076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574690.268082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff920a52890380 x1875444569870592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.268090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569870592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.268095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.268097:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574690.268099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574690.268110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574690.268122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574690.268128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff920a52893480 x1875444569870720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.268136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569870720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.268141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.268143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574690.268145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574690.268148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574690.268158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574690.268163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574690/real 1788574690] req@ffff920a52891880 x1875444569870848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574690.268187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569870848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574690.268196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574690.268200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574690.268203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569870848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574690.715742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574691.291731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574691.739753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574692.315721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574692.763734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574693.339720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574693.787821:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574694.363675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574694.811798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574695.387684:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574695.387689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574695.387692:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574695.387708:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.387718:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11448 00000100:00080000:0.0:1788574695.387727:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.387734:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574695.387773:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569870976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.387797:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.387800:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574695.387802:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.387805:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11448 00000100:00080000:0.0:1788574695.387808:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.387810:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574695.387819:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574695.387836:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569871104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.387844:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.387846:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574695.387850:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574695.387851:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569870976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574695.387855:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11448 00000100:00080000:0.0:1788574695.387861:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.387866:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574695.387870:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574695.387880:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569871232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.387887:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574695.387889:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574695.387895:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.387898:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11448 00000100:00080000:0.0:1788574695.387901:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.387903:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574695.387907:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574695.387917:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569871360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574695.387924:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788574695.387927:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574695.387928:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574695.387931:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574695.387936:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11448 00000100:00080000:0.0:1788574695.387939:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574695.387941:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574695.387944:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574695.387957:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569871488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574695.387979:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff920a48353800 x1875444569870976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.387995:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569870976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.388003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.388006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574695.388009:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569870976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574695.388027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574695.388051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574695.388064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff920a48351c00 x1875444569871104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.388077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569871104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.388086:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.388090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574695.388093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574695.388098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574695.388120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574695.388131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff920a48350a80 x1875444569871232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.388150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569871232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.388159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.388162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574695.388166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574695.388172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574695.388194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574695.388203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff920a48353480 x1875444569871360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.388210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569871360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.388216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.388218:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574695.388220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574695.388223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574695.388234:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574695.388240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574695/real 1788574695] req@ffff920a48350700 x1875444569871488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574695.388247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569871488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574695.388256:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574695.388258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574695.388260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788574695.835744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574696.411734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574696.859743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574697.435689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788574697.883750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574698.459694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574698.907841:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574699.483658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574699.931724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574700.507736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574700.507759:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574700.507768:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574700.507773:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574700.507786:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11453 00000100:00080000:0.0:1788574700.507797:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574700.507805:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574700.507860:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569871616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574700.507888:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574700.507891:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574700.507893:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574700.507897:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11453 00000100:00080000:0.0:1788574700.507900:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574700.507927:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574700.507938:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574700.507938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569871616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574700.507955:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569871744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574700.507969:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574700.507972:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574700.507973:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574700.507977:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11453 00000100:00080000:0.0:1788574700.507980:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574700.507982:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574700.507986:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574700.507994:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569871872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574700.508001:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574700.508004:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574700.508006:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574700.508011:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11453 00000800:00000100:2.0:1788574700.508011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574700.508022:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574700.508025:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574700.508030:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574700.508040:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569872000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574700.508048:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574700.508051:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574700.508053:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574700.508058:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11453 00000100:00080000:0.0:1788574700.508064:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574700.508072:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574700.508073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff920a48350380 x1875444569871616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574700.508083:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574700.508102:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569872128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574700.508110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569871616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574700.508128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574700.508132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574700.508138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569871616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574700.508160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574700.508194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574700.508207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff920a48353b80 x1875444569871744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574700.508222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569871744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574700.508229:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574700.508233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574700.508236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574700.508241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574700.508265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574700.508294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff920a48351500 x1875444569871872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574700.508311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569871872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574700.508324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574700.508330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574700.508334:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569871872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574700.508342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574700.508371:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574700.508388:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff920a48350700 x1875444569872000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574700.508398:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569872000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574700.508407:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574700.508412:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574700.508417:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574700.508426:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574700.508471:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574700.508490:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574700/real 1788574700] req@ffff920a48353480 x1875444569872128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574700.508510:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569872128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574700.508524:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574700.508529:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574700.508537:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574700.955749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574701.531677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574701.979817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574702.555680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574703.003795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574703.579717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574704.027738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574704.603694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574705.051802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574705.627730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574705.627730:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574705.627745:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574705.627751:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574705.627763:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11459 00000100:00080000:3.0:1788574705.627774:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574705.627782:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574705.627828:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569872256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574705.627854:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574705.627857:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574705.627859:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574705.627862:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11459 00000100:00080000:3.0:1788574705.627865:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574705.627867:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574705.627877:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574705.627888:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569872384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574705.627897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574705.627901:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574705.627907:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574705.627908:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569872256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574705.627913:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11459 00000100:00080000:3.0:1788574705.627919:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574705.627927:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574705.627933:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574705.627949:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569872512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574705.627959:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574705.627963:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574705.627966:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574705.627972:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11459 00000100:00080000:3.0:1788574705.627987:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574705.627991:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574705.627997:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574705.627998:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574705.628013:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569872640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574705.628022:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574705.628025:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574705.628027:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574705.628031:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11459 00000100:00080000:3.0:1788574705.628035:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574705.628039:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574705.628044:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574705.628051:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574705/real 1788574705] req@ffff920a529b0e00 x1875444569872256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574705.628062:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569872768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574705.628091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569872256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574705.628106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574705.628111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574705.628116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569872256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574705.628142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574705.628168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574705.628181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574705/real 1788574705] req@ffff920a529b1180 x1875444569872384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574705.628194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569872384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574705.628202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574705.628206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574705.628209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574705.628216:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574705.628236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574705.628246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574705/real 1788574705] req@ffff920a529b1500 x1875444569872512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574705.628258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569872512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574705.628266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574705.628269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574705.628273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574705.628279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574705.628298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574705.628309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574705/real 1788574705] req@ffff920a529b3b80 x1875444569872640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574705.628321:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569872640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574705.628329:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574705.628332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574705.628336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574705.628342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574705.628362:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574705.628373:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574705/real 1788574705] req@ffff920a529b0380 x1875444569872768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574705.628385:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569872768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574705.628394:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574705.628397:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574705.628401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569872768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574706.075782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574706.651730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574707.099754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574707.675745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574708.123798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574708.699671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574709.147736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574709.723697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574710.171716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574710.747706:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574710.747718:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574710.747723:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574710.747734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574710.747735:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11464 00000100:00080000:2.0:1788574710.747759:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574710.747770:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574710.747808:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569872896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574710.747834:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574710.747837:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574710.747838:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574710.747842:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11464 00000100:00080000:2.0:1788574710.747844:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574710.747847:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574710.747855:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574710.747864:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569873024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574710.747872:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574710.747875:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574710.747876:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574710.747882:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11464 00000100:00100000:0.0:1788574710.747884:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569872896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574710.747884:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574710.747891:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574710.747896:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574710.747905:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569873152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574710.747917:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574710.747920:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574710.747922:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574710.747924:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11464 00000100:00080000:2.0:1788574710.747927:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574710.747929:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574710.747933:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574710.747940:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569873280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574710.747946:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788574710.747952:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574710.747952:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574710.747956:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574710.747960:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11464 00000100:00080000:2.0:1788574710.747963:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574710.747965:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574710.747969:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574710.748009:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569873408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574710.748010:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574710/real 1788574710] req@ffff920a60ac6300 x1875444569872896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574710.748047:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569872896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574710.748056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574710.748059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574710.748063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569872896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574710.748068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574710.748085:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574710.748094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574710/real 1788574710] req@ffff920a60ac5500 x1875444569873024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574710.748102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569873024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574710.748108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574710.748117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574710.748120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574710.748140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574710.748152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574710.748159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574710/real 1788574710] req@ffff920a60ac4000 x1875444569873152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574710.748167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569873152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574710.748172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574710.748174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574710.748177:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574710.748180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574710.748190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574710.748196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574710/real 1788574710] req@ffff920a60ac5c00 x1875444569873280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574710.748203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569873280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574710.748208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574710.748209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574710.748211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574710.748214:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574710.748223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574710.748229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574710/real 1788574710] req@ffff920a60ac4380 x1875444569873408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574710.748236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569873408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574710.748241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574710.748243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574710.748245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574711.195756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574711.771677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574712.219705:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574712.795693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574713.243797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574713.819661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574714.267758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574714.843684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574715.291776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574715.867674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574715.867700:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574715.867708:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574715.867713:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574715.867726:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11469 00000100:00080000:0.0:1788574715.867739:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574715.867749:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574715.867793:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569873536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574715.867826:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574715.867830:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574715.867833:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574715.867839:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11469 00000100:00080000:0.0:1788574715.867845:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574715.867851:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574715.867864:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574715.867885:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569873664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574715.867891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569873536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574715.867899:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574715.867903:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574715.867906:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574715.867910:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11469 00000100:00080000:0.0:1788574715.867914:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574715.867918:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574715.867922:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574715.867932:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569873792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574715.867938:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574715.867943:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574715.867944:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574715.867948:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11469 00000100:00080000:0.0:1788574715.867950:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574715.867953:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574715.867957:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574715.867967:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574715.867969:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569873920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574715.867979:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574715.867983:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574715.867984:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574715.867991:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11469 00000100:00080000:0.0:1788574715.867999:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574715.868005:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574715.868012:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574715.868012:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574715/real 1788574715] req@ffff920a48350a80 x1875444569873536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574715.868036:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569874048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574715.868044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569873536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574715.868058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574715.868064:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574715.868068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569873536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574715.868090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574715.868107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574715.868116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574715/real 1788574715] req@ffff920a48351c00 x1875444569873664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574715.868124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569873664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574715.868130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574715.868132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574715.868134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574715.868138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574715.868148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574715.868154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574715/real 1788574715] req@ffff920a48353800 x1875444569873792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574715.868162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569873792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574715.868172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574715.868174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574715.868176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574715.868179:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574715.868191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574715.868199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574715/real 1788574715] req@ffff920a48353480 x1875444569873920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574715.868211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569873920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574715.868219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574715.868223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574715.868226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569873920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574715.868233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574715.868255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574715.868267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574715/real 1788574715] req@ffff920a48350700 x1875444569874048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574715.868277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569874048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574715.868282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574715.868284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574715.868287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574716.315787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574716.891717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574717.339734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574717.915740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574718.363799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574718.939705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574719.387720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574719.963732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574720.411817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574720.987680:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574720.987692:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788574720.987696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574720.987697:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574720.987716:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11474 00000100:00080000:3.0:1788574720.987730:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574720.987741:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574720.987790:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569874176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574720.987819:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574720.987822:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574720.987823:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574720.987827:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11474 00000100:00080000:3.0:1788574720.987830:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574720.987834:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574720.987848:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574720.987863:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569874304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574720.987877:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574720.987878:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569874176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574720.987886:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574720.987889:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574720.987893:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11474 00000100:00080000:3.0:1788574720.987897:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574720.987900:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574720.987905:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574720.987915:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569874432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574720.987922:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574720.987925:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574720.987926:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574720.987929:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11474 00000100:00080000:3.0:1788574720.987947:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574720.987949:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574720.987953:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574720.987960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574720.987963:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569874560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574720.987972:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574720.987976:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574720.987979:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574720.987984:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11474 00000100:00080000:3.0:1788574720.987988:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574720.987993:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574720.987999:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574720.988002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574720/real 1788574720] req@ffff920a529b2300 x1875444569874176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574720.988015:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569874688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574720.988022:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569874176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574720.988039:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574720.988043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574720.988046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569874176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574720.988087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574720.988109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574720.988118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574720/real 1788574720] req@ffff920a529b0700 x1875444569874304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574720.988126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569874304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574720.988132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574720.988135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574720.988137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574720.988140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574720.988151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574720.988157:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574720/real 1788574720] req@ffff920a529b0000 x1875444569874432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574720.988165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569874432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574720.988174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574720.988178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574720.988181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574720.988188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574720.988211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574720.988223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574720/real 1788574720] req@ffff920a529b1f80 x1875444569874560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574720.988235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569874560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574720.988240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574720.988243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574720.988245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574720.988249:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574720.988264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574720.988270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574720/real 1788574720] req@ffff920a529b3100 x1875444569874688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574720.988278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569874688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574745 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574720.988283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574720.988286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574720.988288:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574721.435774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574722.011738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574722.459843:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574723.035682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574723.483734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574724.059694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574724.507841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574725.083691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574725.531745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574726.107712:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574726.107717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574726.107722:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574726.107727:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.107747:0:167407:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11479 00000100:00080000:0.0:1788574726.107756:0:167407:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.107763:0:167407:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574726.107795:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569874816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.107818:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.107821:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574726.107823:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.107826:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11479 00000100:00080000:0.0:1788574726.107829:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.107831:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574726.107839:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574726.107849:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569874944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.107858:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.107861:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574726.107862:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.107868:0:167407:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11479 00000100:00100000:2.0:1788574726.107870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569874816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574726.107871:0:167407:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.107877:0:167407:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574726.107881:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574726.107890:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569875072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574726.107897:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.107900:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574726.107901:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.107905:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11479 00000100:00080000:0.0:1788574726.107907:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.107909:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574726.107913:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574726.107926:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569875200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574726.107932:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574726.107940:0:167407:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574726.107944:0:167407:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574726.107945:0:167407:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574726.107950:0:167407:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11479 00000100:00080000:0.0:1788574726.107953:0:167407:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574726.107956:0:167407:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574726.107961:0:167407:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574726.107966:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574726] req@ffff920a48351500 x1875444569874816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574726.107972:0:167407:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569875328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574726.107993:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569874816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574726.108003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574726.108006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574726.108009:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569874816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574726.108014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574726.108028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574726.108036:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574726] req@ffff920a48353b80 x1875444569874944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574726.108043:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569874944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574726.108049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574726.108051:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574726.108053:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569874944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574726.108071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574726.108082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574726.108088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574726] req@ffff920a48350380 x1875444569875072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574726.108096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569875072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574726.108100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574726.108102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574726.108104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574726.108107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574726.108117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574726.108122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574726] req@ffff920a48350700 x1875444569875200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574726.108129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569875200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574726.108134:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574726.108136:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574726.108138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574726.108141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574726.108150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574726.108155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574726/real 1788574726] req@ffff920a48353480 x1875444569875328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574726.108163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569875328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574726.108168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574726.108170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574726.108171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574726.555745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574727.131739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574727.579718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574728.155680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574728.603746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574729.179703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574729.627737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574730.203682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574730.651772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574731.227713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574731.227715:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.227727:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574731.227734:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.227750:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11484 00000100:00080000:2.0:1788574731.227764:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.227773:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574731.227828:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569875456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.227858:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.227862:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574731.227864:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.227868:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11484 00000100:00080000:2.0:1788574731.227871:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.227875:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574731.227887:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574731.227899:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569875584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574731.227907:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569875456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574731.227925:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.227928:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574731.227930:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.227936:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11484 00000100:00080000:2.0:1788574731.227939:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.227943:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574731.227948:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574731.227961:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569875712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.227972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574731.227976:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574731.227977:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.227984:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11484 00000800:00000100:0.0:1788574731.227985:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574731.227991:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574731.227996:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574731.228002:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574731.228014:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569875840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574731.228024:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574731.228028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff920a60ac4e00 x1875444569875456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574731.228029:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574731.228032:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574731.228042:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11484 00000100:00080000:2.0:1788574731.228048:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574731.228053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569875456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574731.228057:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788574731.228071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788574731.228074:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574731.228076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574731.228083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569875456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574731.228093:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569875968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574731.228109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574731.228157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574731.228168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff920a60ac7800 x1875444569875584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.228183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569875584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.228189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.228191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574731.228194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574731.228198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574731.228218:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574731.228229:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff920a60ac4380 x1875444569875712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.228242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569875712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.228252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.228256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574731.228259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574731.228264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574731.228282:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574731.228294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff920a60ac5c00 x1875444569875840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.228306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569875840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.228321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.228325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574731.228329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574731.228336:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574731.228353:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574731.228361:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574731/real 1788574731] req@ffff920a60ac4000 x1875444569875968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574731.228369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569875968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574731.228375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574731.228378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574731.228380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569875968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574731.675711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574732.251697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574732.699797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574733.275721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574733.723760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574734.299715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574734.747760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574735.323698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574735.771759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574736.347719:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.347736:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574736.347741:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.347757:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11489 00000100:00080000:2.0:1788574736.347775:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574736.347782:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574736.347787:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574736.347847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569876096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.347888:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.347893:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574736.347896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.347903:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11489 00000100:00080000:2.0:1788574736.347908:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.347915:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574736.347928:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574736.347931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569876096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574736.347954:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569876224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.347974:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.347979:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574736.347982:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.347993:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11489 00000100:00080000:2.0:1788574736.348001:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788574736.348002:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574736.348014:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574736.348023:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574736.348046:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569876352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574736.348052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff920a60ac5500 x1875444569876096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.348093:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.348098:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574736.348101:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.348108:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11489 00000100:00080000:2.0:1788574736.348112:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574736.348118:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574736.348125:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574736.348140:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569876480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574736.348153:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574736.348162:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574736.348169:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574736.348177:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11489 00000100:00080000:2.0:1788574736.348183:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574736.348190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569876096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574736.348197:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574736.348209:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788574736.348216:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.348224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574736.348230:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569876608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574736.348231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569876096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574736.348371:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574736.348489:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574736.348509:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff920a60ac6300 x1875444569876224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.348525:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569876224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.348535:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.348539:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574736.348543:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574736.348551:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574736.348574:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574736.348620:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff920a60ac7100 x1875444569876352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.348635:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569876352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.348646:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.348649:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574736.348652:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574736.348658:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574736.348691:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574736.348700:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff920a60ac5180 x1875444569876480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.348708:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569876480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.348714:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.348717:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574736.348719:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574736.348723:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574736.348735:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574736.348744:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574736/real 1788574736] req@ffff920a60ac4a80 x1875444569876608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574736.348756:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569876608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574736.348765:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574736.348768:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574736.348772:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574736.795806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574737.371779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574737.819786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574738.395682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574738.843780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574739.419696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574739.867786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574740.443712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574740.891762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574741.467716:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574741.467717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574741.467729:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574741.467736:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574741.467750:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11494 00000100:00080000:2.0:1788574741.467763:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574741.467773:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574741.467825:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569876736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574741.467859:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574741.467863:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574741.467865:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574741.467870:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11494 00000100:00080000:2.0:1788574741.467875:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574741.467879:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574741.467891:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574741.467907:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569876864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574741.467911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569876736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574741.467928:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574741.467931:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574741.467933:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574741.467940:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11494 00000100:00080000:2.0:1788574741.467944:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574741.467949:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574741.467955:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574741.467969:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569876992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574741.467979:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574741.467983:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574741.467984:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574741.467989:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11494 00000100:00080000:2.0:1788574741.467993:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574741.467997:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574741.468004:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574741.468016:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569877120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574741.468026:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876864:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788574741.468030:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574741.468036:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574741.468038:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574741.468043:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11494 00000100:00100000:3.0:1788574741.468058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876992:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788574741.468059:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574741.468068:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574741.468076:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574741.468080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574741.468092:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569877248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574741.468130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574741.915758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574742.491772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574742.939773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574743.515746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574743.963836:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574744.539697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574744.540061:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574744.540129:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574744.540153:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574744.540229:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574744.540255:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574744.540265:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574744.540274:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574744.540379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574744] req@ffff920a60ac7100 x1875444569877248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574744.540411:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569877248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574744.540425:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574744.540431:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574744.540437:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877248:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574744.540447:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574744] req@ffff920a60ac5180 x1875444569877120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574744.540462:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569877120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574744.540472:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574744.540476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574744.540479:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574744.540487:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574744] req@ffff920a60ac6a00 x1875444569876864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574744.540501:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569876864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574744.540510:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574744.540514:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574744.540518:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574744.540528:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574744] req@ffff920a60ac4a80 x1875444569876992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574744.540540:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569876992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574744.540549:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574744.540553:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574744.540557:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569876992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574744.540564:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574741/real 1788574744] req@ffff920a60ac4700 x1875444569876736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574744.540578:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569876736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574744.540621:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574744.540626:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574744.540630:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569876736:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788574744.987770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574745.563702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574746.011802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574746.587759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574746.587831:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574746.587836:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574746.587840:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574746.587851:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11499 00000100:00080000:0.0:1788574746.587861:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574746.587870:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574746.587910:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569877376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574746.587937:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574746.587940:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574746.587942:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574746.587945:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11499 00000100:00080000:0.0:1788574746.587949:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574746.587952:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574746.587961:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574746.587972:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569877504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574746.587981:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574746.587982:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569877376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574746.587985:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574746.587989:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574746.587996:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11499 00000100:00080000:0.0:1788574746.587999:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574746.588003:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574746.588009:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574746.588023:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569877632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574746.588037:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574746.588041:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574746.588043:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574746.588046:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11499 00000100:00080000:0.0:1788574746.588050:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574746.588054:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574746.588059:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574746.588071:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574746.588075:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569877760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574746.588087:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574746.588091:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574746.588093:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574746.588098:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11499 00000100:00080000:0.0:1788574746.588102:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574746.588106:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574746.588112:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574746.588117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff920a48353800 x1875444569877376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574746.588128:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569877888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574746.588145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569877376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.588165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.588170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574746.588175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569877376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574746.588202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574746.588246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574746.588262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff920a48351c00 x1875444569877504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.588287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569877504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.588296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.588299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574746.588302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574746.588308:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574746.588330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574746.588342:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff920a48350a80 x1875444569877632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.588353:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569877632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.588361:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.588364:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574746.588367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574746.588374:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574746.588396:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574746.588407:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff920a48353480 x1875444569877760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.588420:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569877760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.588428:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.588432:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574746.588436:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574746.588442:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574746.588465:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574746.588476:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574746/real 1788574746] req@ffff920a48350700 x1875444569877888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574746.588487:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569877888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574746.588495:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574746.588498:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574746.588502:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569877888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574747.035796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574747.611848:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574748.059809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574748.635716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574749.083740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574749.659745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574750.107736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574750.683689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574751.131752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574751.707719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574751.707748:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574751.707765:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574751.707771:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574751.707784:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11505 00000100:00080000:0.0:1788574751.707799:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574751.707808:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574751.707863:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569878016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574751.707897:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574751.707901:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574751.707904:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574751.707908:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11505 00000100:00080000:0.0:1788574751.707913:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574751.707918:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574751.707930:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574751.707944:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569878016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574751.707951:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569878144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574751.707968:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574751.707973:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574751.707976:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574751.707981:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11505 00000100:00080000:0.0:1788574751.707985:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574751.707991:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574751.707997:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574751.708009:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569878272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788574751.708018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574751.708043:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574751.708048:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574751.708050:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574751.708058:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11505 00000100:00080000:0.0:1788574751.708062:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574751.708066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574751/real 1788574751] req@ffff920a48350380 x1875444569878016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574751.708071:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574751.708089:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574751.708099:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569878016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574751.708108:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569878400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574751.708109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574751.708119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574751.708131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569878016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574751.708148:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574751.708152:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574751.708154:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574751.708161:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11505 00000100:00080000:0.0:1788574751.708167:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574751.708172:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574751.708177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878144:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788574751.708192:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574751.708204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574751.708213:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569878528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574751.708223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574751/real 1788574751] req@ffff920a48353b80 x1875444569878144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574751.708269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569878144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574751.708276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574751.708280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574751.708283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574751.708288:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574751.708303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574751.708310:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574751/real 1788574751] req@ffff920a48351500 x1875444569878272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574751.708322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569878272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574751.708327:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574751.708328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574751.708330:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574751.708334:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574751.708345:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574751.708351:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574751/real 1788574751] req@ffff920a48350700 x1875444569878400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574751.708359:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569878400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574751.708363:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574751.708365:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574751.708367:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574751.708377:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574751.708388:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574751.708397:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574751/real 1788574751] req@ffff920a48353480 x1875444569878528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574751.708408:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569878528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574751.708439:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574751.708443:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574751.708447:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574752.155788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574752.731718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574753.179869:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574753.755706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574754.203766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574754.779690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574755.227798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574755.803715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574756.251790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574756.827714:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574756.827715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574756.827728:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574756.827734:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574756.827746:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11510 00000100:00080000:0.0:1788574756.827769:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574756.827780:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574756.827835:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569878656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574756.827869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574756.827872:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574756.827875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574756.827879:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11510 00000100:00080000:0.0:1788574756.827884:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574756.827889:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574756.827902:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569878656:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574756.827903:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574756.827923:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569878784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574756.827934:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574756.827938:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574756.827940:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574756.827944:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11510 00000100:00080000:0.0:1788574756.827948:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574756.827963:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574756.827969:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788574756.827978:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574756.827984:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569878912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574756.827995:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574756.828000:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574756.828002:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574756.828006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11510 00000100:00080000:0.0:1788574756.828011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574756.828016:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574756.828022:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574756.828023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574756/real 1788574756] req@ffff920a48350a80 x1875444569878656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574756.828039:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569879040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574756.828049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569878656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574756.828051:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574756.828061:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574756.828066:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574756.828070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574756.828073:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11510 00000100:00080000:3.0:1788574756.828077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788574756.828080:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788574756.828083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569878656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574756.828088:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574756.828095:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574756.828109:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569879168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574756.828112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574756.828141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574756.828154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574756/real 1788574756] req@ffff920a48351c00 x1875444569878784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574756.828171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569878784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574756.828178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574756.828181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574756.828183:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574756.828188:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574756.828200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574756.828207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574756/real 1788574756] req@ffff920a48353800 x1875444569878912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574756.828215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569878912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574756.828220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574756.828223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574756.828225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569878912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574756.828228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574756.828239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574756.828244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574756/real 1788574756] req@ffff920a48353480 x1875444569879040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574756.828251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569879040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574756.828256:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574756.828258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574756.828261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574756.828288:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574756.828303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574756.828311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574756/real 1788574756] req@ffff920a48350700 x1875444569879168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574756.828322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569879168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574756.828331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574756.828335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574756.828339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574757.275780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574757.851725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574758.299755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574758.875688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574759.323779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574759.899656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574760.347756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574760.923744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574761.371772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574761.947714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574761.947723:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574761.947731:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574761.947735:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574761.947745:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11515 00000100:00080000:0.0:1788574761.947757:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574761.947772:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574761.947819:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569879296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574761.947842:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574761.947845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574761.947846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574761.947849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11515 00000100:00080000:0.0:1788574761.947851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574761.947854:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574761.947863:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574761.947872:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569879424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574761.947880:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574761.947884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574761.947886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788574761.947886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569879296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574761.947894:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11515 00000100:00080000:0.0:1788574761.947897:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574761.947899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574761.947901:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574761.947909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569879552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574761.947915:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574761.947917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574761.947918:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574761.947920:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11515 00000100:00080000:0.0:1788574761.947923:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574761.947926:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574761.947930:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574761.947938:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569879680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574761.947943:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574761.947945:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574761.947946:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574761.947949:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11515 00000100:00080000:0.0:1788574761.947951:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574761.947954:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574761.947956:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788574761.947961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574761.947965:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569879808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574761.948009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574761/real 1788574761] req@ffff920a48351500 x1875444569879296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574761.948027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569879296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574761.948038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574761.948041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574761.948046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569879296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574761.948068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574761.948084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574761.948094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574761/real 1788574761] req@ffff920a48353b80 x1875444569879424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574761.948102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569879424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574761.948130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574761.948134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574761.948137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574761.948141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574761.948156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574761.948164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574761/real 1788574761] req@ffff920a48350380 x1875444569879552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574761.948172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569879552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574761.948177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574761.948180:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574761.948182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574761.948185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574761.948196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574761.948201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574761/real 1788574761] req@ffff920a48350700 x1875444569879680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574761.948209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569879680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574761.948214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574761.948217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574761.948219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574761.948222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574761.948232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574761.948237:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574761/real 1788574761] req@ffff920a48353480 x1875444569879808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574761.948244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569879808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574786 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574761.948248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574761.948251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574761.948253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569879808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574762.395720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574762.971739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574763.419851:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574763.995710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574764.443787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574765.019697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574765.467788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574766.043746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574766.491761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574767.067718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574767.067732:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574767.067739:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574767.067745:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574767.067757:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11520 00000100:00080000:3.0:1788574767.067770:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574767.067780:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574767.067825:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569879936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574767.067852:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574767.067855:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574767.067857:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574767.067860:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11520 00000100:00080000:3.0:1788574767.067863:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574767.067867:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574767.067878:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574767.067886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569879936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574767.067896:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569880064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574767.067907:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574767.067911:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574767.067913:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574767.067918:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11520 00000100:00080000:3.0:1788574767.067922:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574767.067926:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574767.067932:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574767.067947:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569880192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574767.067983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574767.067992:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574767.067995:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574767.067997:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574767.068001:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11520 00000100:00080000:3.0:1788574767.068005:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574767.068009:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574767.068015:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574767.068026:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569880320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574767.068033:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574767.068035:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574767.068036:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574767.068039:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11520 00000100:00080000:3.0:1788574767.068041:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574767.068044:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574767.068048:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574767.068049:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff920a529b2d80 x1875444569879936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574767.068059:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569880448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574767.068082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569879936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574767.068097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574767.068101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574767.068106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569879936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574767.068135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574767.068156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574767.068167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff920a529b3480 x1875444569880064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574767.068176:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569880064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574767.068182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574767.068184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574767.068187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574767.068191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574767.068202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574767.068208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff920a529b2a00 x1875444569880192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574767.068216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569880192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574767.068221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574767.068223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574767.068225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574767.068228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574767.068240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574767.068252:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff920a529b1c00 x1875444569880320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574767.068261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569880320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574767.068266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574767.068268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574767.068270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574767.068273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574767.068284:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574767.068290:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574767/real 1788574767] req@ffff920a529b3100 x1875444569880448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574767.068297:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569880448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574767.068302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574767.068304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574767.068306:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574767.515817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574768.091717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574768.539850:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574769.115755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574769.563822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574770.139704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574770.587795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574771.163692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574771.611749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574772.187691:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.187703:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574772.187708:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.187728:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11525 00000100:00080000:3.0:1788574772.187739:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574772.187742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574772.187747:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574772.187783:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569880576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.187814:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.187819:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574772.187821:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.187826:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11525 00000100:00080000:3.0:1788574772.187831:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.187837:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574772.187840:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569880576:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574772.187848:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574772.187865:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569880704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.187874:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.187878:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574772.187880:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.187884:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11525 00000100:00080000:3.0:1788574772.187888:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.187892:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574772.187898:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574772.187910:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569880832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574772.187918:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574772.187925:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.187929:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574772.187931:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574772.187936:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11525 00000100:00080000:3.0:1788574772.187942:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.187946:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574772.187952:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574772.187959:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff920a529b1f80 x1875444569880576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574772.187965:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569880960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574772.187989:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574772.187992:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574772.187993:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788574772.187997:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569880576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574772.187999:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11525 00000100:00080000:3.0:1788574772.188004:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574772.188010:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788574772.188011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.188018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:3.0:1788574772.188018:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574772.188024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569880576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574772.188044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574772.188063:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574772.188071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff920a529b0000 x1875444569880704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.188078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569880704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.188084:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.188086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574772.188088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574772.188099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574772.188111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574772.188116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff920a529b0700 x1875444569880832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.188123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569880832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.188128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.188130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574772.188132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574772.188135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574772.188150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574772.188161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff920a529b2300 x1875444569880960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574772.188171:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569881088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574772.188177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569880960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.188191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.188195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574772.188199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569880960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574772.188211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574772.188225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574772.188232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574772/real 1788574772] req@ffff920a529b0380 x1875444569881088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574772.188239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569881088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574772.188245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574772.188247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574772.188250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574772.635828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574773.211696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574773.659729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788574774.235721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574774.683815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574775.259729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574775.707733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574776.283759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574776.731789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574777.307712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574777.307742:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574777.307749:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574777.307754:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574777.307767:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11530 00000100:00080000:0.0:1788574777.307780:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574777.307789:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574777.307836:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569881216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574777.307867:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574777.307870:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574777.307872:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574777.307875:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11530 00000100:00080000:0.0:1788574777.307878:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574777.307881:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574777.307893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574777.307905:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569881344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574777.307918:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569881216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574777.307922:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574777.307927:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574777.307929:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574777.307934:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11530 00000100:00080000:0.0:1788574777.307938:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574777.307942:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574777.307947:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574777.307964:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569881472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574777.307973:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574777.307976:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574777.307979:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574777.307982:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11530 00000100:00080000:0.0:1788574777.307986:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574777.307991:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574777.307996:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574777.308010:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569881600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574777.308013:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574777.308023:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574777.308026:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574777.308028:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574777.308033:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11530 00000100:00080000:0.0:1788574777.308037:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574777.308042:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574777.308047:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574777.308055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff920a48353800 x1875444569881216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574777.308059:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569881728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574777.308078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569881216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.308088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.308093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574777.308096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569881216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574777.308118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574777.308132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574777.308140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff920a48351c00 x1875444569881344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.308148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569881344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.308153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.308155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574777.308157:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574777.308160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574777.308169:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574777.308175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff920a48350a80 x1875444569881472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.308193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569881472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.308201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.308204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574777.308207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574777.308213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574777.308242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574777.308253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff920a48353480 x1875444569881600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.308266:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569881600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.308273:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.308276:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574777.308280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574777.308285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574777.308308:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574777.308318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574777/real 1788574777] req@ffff920a48350700 x1875444569881728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574777.308339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569881728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574777.308353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574777.308356:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574777.308366:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574777.755773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574778.331952:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574778.779771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574779.355692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574779.803802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574780.379700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574780.827831:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574781.403694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574781.851809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574782.428568:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574782.428615:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574782.428622:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574782.428625:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574782.428639:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11535 00000100:00080000:3.0:1788574782.428649:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574782.428656:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574782.428689:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569881856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574782.428716:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574782.428720:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574782.428722:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574782.428726:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11535 00000100:00080000:3.0:1788574782.428729:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574782.428734:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574782.428744:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574782.428751:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569881856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574782.428757:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569881984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574782.428767:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574782.428771:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574782.428772:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574782.428776:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11535 00000100:00080000:3.0:1788574782.428779:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574782.428783:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574782.428788:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574782.428798:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569882112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574782.428807:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574782.428809:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574782.428814:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574782.428815:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574782.428820:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11535 00000100:00080000:3.0:1788574782.428825:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574782.428829:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574782.428834:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574782.428837:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff920a529b3b80 x1875444569881856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574782.428846:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569882240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574782.428853:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569881856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574782.428860:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574782.428864:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574782.428865:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574782.428868:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574782.428870:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574782.428875:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11535 00000100:00100000:1.0:1788574782.428876:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569881856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574782.428879:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574782.428884:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574782.428889:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574782.428895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574782.428901:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569882368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574782.428914:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574782.428923:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff920a529b1500 x1875444569881984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.428932:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569881984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.428938:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.428940:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574782.428942:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569881984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574782.428945:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574782.428955:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574782.428960:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff920a529b1180 x1875444569882112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.428966:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569882112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.428971:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.428973:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574782.428974:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574782.428978:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574782.428987:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574782.428992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff920a529b0e00 x1875444569882240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.428999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569882240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.429003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.429005:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574782.429007:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574782.429017:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574782.429033:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574782.429042:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574782/real 1788574782] req@ffff920a529b3b80 x1875444569882368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574782.429052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569882368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574782.429059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574782.429062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574782.429065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574782.875793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574783.451705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574783.899774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574784.475706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574784.923765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574785.499657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574785.947714:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574786.523921:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574786.971783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788574787.547674:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574787.547678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574787.547692:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574787.547699:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574787.547709:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11540 00000100:00080000:2.0:1788574787.547720:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574787.547727:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574787.547761:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569882496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574787.547784:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574787.547788:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574787.547790:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574787.547793:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11540 00000100:00080000:2.0:1788574787.547797:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574787.547801:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574787.547810:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574787.547817:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569882496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574787.547824:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569882624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574787.547833:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574787.547837:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574787.547838:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574787.547843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11540 00000100:00080000:2.0:1788574787.547846:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574787.547849:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574787.547853:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574787.547863:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569882752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574787.547870:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574787.547873:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574787.547875:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574787.547880:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574787.547880:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11540 00000100:00080000:2.0:1788574787.547887:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574787.547890:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574787.547895:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574787.547905:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569882880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574787.547913:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574787.547916:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574787.547917:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574787.547923:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11540 00000100:00100000:1.0:1788574787.547924:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff920a60ac6300 x1875444569882496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574787.547927:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574787.547934:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574787.547942:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574787.547949:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569882496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574787.547954:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569883008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574787.547981:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.547985:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574787.547990:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569882496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574787.548018:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574787.548038:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574787.548050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff920a60ac5500 x1875444569882624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.548062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569882624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.548069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.548072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574787.548075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574787.548081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574787.548098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574787.548108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff920a60ac4000 x1875444569882752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.548119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569882752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.548127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.548130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574787.548133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574787.548138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574787.548160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574787.548173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff920a60ac5c00 x1875444569882880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.548185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569882880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.548192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.548195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574787.548198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569882880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574787.548204:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574787.548221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574787.548231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574787/real 1788574787] req@ffff920a60ac4380 x1875444569883008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574787.548253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569883008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574787.548261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574787.548264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574787.548267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574787.995687:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574788.571695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574789.019780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574789.595773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574790.043728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574790.619685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574791.067739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574791.643693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574792.091744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574792.667661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574792.667661:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574792.667670:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574792.667673:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574792.667682:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11546 00000100:00080000:3.0:1788574792.667690:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574792.667696:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574792.667729:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569883136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574792.667748:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574792.667750:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574792.667752:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574792.667755:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11546 00000100:00080000:3.0:1788574792.667757:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574792.667760:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574792.667767:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574792.667778:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569883264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574792.667779:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569883136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574792.667798:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574792.667801:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574792.667804:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574792.667808:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11546 00000100:00080000:3.0:1788574792.667812:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574792.667816:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574792.667821:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574792.667833:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569883392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574792.667836:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574792.667844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574792.667847:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574792.667848:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574792.667851:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11546 00000100:00080000:3.0:1788574792.667854:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574792.667857:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574792.667861:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574792.667869:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569883520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574792.667875:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574792/real 1788574792] req@ffff920a529b0380 x1875444569883136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574792.667876:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574792.667883:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574792.667885:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574792.667891:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11546 00000100:00080000:3.0:1788574792.667895:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574792.667897:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574792.667900:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574792.667902:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569883136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574792.667909:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569883648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574792.667912:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574792.667919:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574792.667923:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569883136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574792.667940:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574792.667959:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574792.667966:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574792/real 1788574792] req@ffff920a529b2300 x1875444569883264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574792.667974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569883264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574792.667979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574792.667981:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574792.667983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574792.667986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574792.667995:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574792.668001:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574792/real 1788574792] req@ffff920a529b0700 x1875444569883392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574792.668007:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569883392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574792.668012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574792.668013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574792.668015:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574792.668018:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574792.668027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574792.668032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574792/real 1788574792] req@ffff920a529b0000 x1875444569883520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574792.668039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569883520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574792.668043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574792.668045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574792.668046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574792.668049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574792.668057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574792.668062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574792/real 1788574792] req@ffff920a529b1f80 x1875444569883648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574792.668073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569883648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574792.668078:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574792.668080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574792.668082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574793.115769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574793.691725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574794.139766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574794.715721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574795.163757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574795.739687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574796.187784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574796.763652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574797.211823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574797.787726:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574797.787726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574797.787737:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574797.787742:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574797.787754:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11551 00000100:00080000:0.0:1788574797.787767:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574797.787777:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574797.787825:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569883776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574797.787855:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574797.787858:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574797.787860:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574797.787863:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11551 00000100:00080000:0.0:1788574797.787867:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574797.787870:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574797.787881:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574797.787886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569883776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574797.787894:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569883904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574797.787902:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574797.787904:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574797.787905:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574797.787909:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11551 00000100:00080000:0.0:1788574797.787911:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574797.787913:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574797.787916:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574797.787926:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569884032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574797.787933:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574797.787936:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574797.787937:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574797.787939:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11551 00000100:00080000:0.0:1788574797.787949:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574797.787954:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574797.787959:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574797.787970:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574797.787976:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569884160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574797.787988:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574797.787992:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574797.787994:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574797.787999:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11551 00000100:00080000:0.0:1788574797.788004:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574797.788009:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574797.788025:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574797.788029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574797/real 1788574797] req@ffff920a48350380 x1875444569883776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574797.788039:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569884288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574797.788050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569883776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574797.788061:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574797.788065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574797.788070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569883776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574797.788100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574797.788125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574797.788139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574797/real 1788574797] req@ffff920a48353b80 x1875444569883904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574797.788154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569883904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574797.788165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574797.788168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574797.788170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569883904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574797.788175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574797.788191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574797.788199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574797/real 1788574797] req@ffff920a48351500 x1875444569884032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574797.788208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569884032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574797.788214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574797.788216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574797.788218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574797.788221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574797.788233:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574797.788239:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574797/real 1788574797] req@ffff920a48350700 x1875444569884160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574797.788245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569884160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574797.788251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574797.788253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574797.788255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574797.788259:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574797.788269:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574797.788275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574797/real 1788574797] req@ffff920a48353480 x1875444569884288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574797.788282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569884288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574797.788290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574797.788293:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574797.788296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574798.235836:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574798.811746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574799.259824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574799.835723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574800.283777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574800.859741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574801.307794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574801.883711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574802.331825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574802.907713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574802.907713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574802.907728:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574802.907733:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574802.907747:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11556 00000100:00080000:1.0:1788574802.907761:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574802.907771:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574802.907824:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569884416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574802.907856:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574802.907874:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574802.907877:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574802.907883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11556 00000100:00080000:1.0:1788574802.907888:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574802.907893:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574802.907909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569884416:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574802.907913:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574802.907931:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569884544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574802.907944:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574802.907947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574802.907949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574802.907954:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11556 00000100:00080000:1.0:1788574802.907958:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574802.907962:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574802.907967:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574802.907981:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569884672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574802.907997:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574802.907998:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574802.908004:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574802.908006:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574802.908012:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11556 00000100:00080000:1.0:1788574802.908017:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574802.908023:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574802.908030:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574802.908037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574802/real 1788574802] req@ffff920a52891f80 x1875444569884416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574802.908046:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569884800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574802.908056:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569884416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574802.908062:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574802.908070:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574802.908072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574802.908073:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574802.908079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788574802.908082:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11556 00000100:00100000:0.0:1788574802.908086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569884416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574802.908089:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574802.908096:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574802.908102:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574802.908112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574802.908118:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569884928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574802.908134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574802.908146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574802/real 1788574802] req@ffff920a52891500 x1875444569884544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574802.908154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569884544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574802.908161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574802.908163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574802.908165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574802.908169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574802.908179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574802.908185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574802/real 1788574802] req@ffff920a52892680 x1875444569884672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574802.908192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569884672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574802.908197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574802.908199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574802.908201:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574802.908205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574802.908226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574802.908233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574802/real 1788574802] req@ffff920a52892a00 x1875444569884800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574802.908241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569884800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574802.908245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574802.908247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574802.908249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574802.908258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574802.908269:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574802.908275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574802/real 1788574802] req@ffff920a52890a80 x1875444569884928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574802.908282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569884928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574802.908287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574802.908289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574802.908291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569884928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574803.355805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574803.931728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574804.379786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574804.955729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574805.403758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574805.979696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574806.427797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574807.003725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574807.451809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574808.027741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574808.027835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.027842:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574808.027847:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.027860:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11561 00000100:00080000:0.0:1788574808.027876:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.027886:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574808.027945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569885056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.027980:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.027984:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.027986:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.027992:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11561 00000100:00080000:0.0:1788574808.027997:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.028002:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574808.028021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569885056:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574808.028025:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.028040:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569885184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.028051:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.028055:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574808.028057:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.028063:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11561 00000100:00080000:0.0:1788574808.028068:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.028073:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574808.028079:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574808.028091:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569885312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.028101:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.028105:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574808.028107:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.028111:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11561 00000100:00080000:0.0:1788574808.028115:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.028119:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574808.028125:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574808.028137:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569885440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574808.028153:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574808.028157:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574808.028159:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574808.028163:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11561 00000100:00100000:1.0:1788574808.028164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885184:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574808.028167:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574808.028172:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574808.028178:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574808.028190:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569885568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574808.028200:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574808.028229:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574808.028262:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574808.475813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574809.051834:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574809.499812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574810.075787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574810.523798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574811.099910:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788574811.100020:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788574811.100072:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788574811.100091:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574811.100136:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574811.100149:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574811.100162:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788574811.100175:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788574811.100177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574811] req@ffff920a48351c00 x1875444569885184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574811.100209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569885184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574811.100224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574811.100230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574811.100235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574811.100243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574811] req@ffff920a48353800 x1875444569885312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574811.100257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569885312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574811.100266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574811.100270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574811.100274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574811.100281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574811] req@ffff920a48353480 x1875444569885440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574811.100295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569885440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574811.100303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574811.100307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574811.100310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574811.100325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574811] req@ffff920a48350a80 x1875444569885056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574811.100340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569885056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574811.100362:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574811.100367:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574811.100370:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569885056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574811.100405:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574808/real 1788574811] req@ffff920a48350700 x1875444569885568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574811.100419:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569885568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574811.100427:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574811.100431:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574811.100434:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574811.547798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574812.123809:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574812.571798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574813.147781:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574813.147797:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574813.147804:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574813.147817:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574813.147819:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11566 00000100:00080000:1.0:1788574813.147841:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574813.147851:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574813.147893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569885696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574813.147924:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574813.147928:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574813.147930:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574813.147934:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11566 00000100:00080000:1.0:1788574813.147938:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574813.147946:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574813.147957:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574813.147981:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569885696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574813.147982:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569885824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574813.148012:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574813.148016:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574813.148018:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574813.148025:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11566 00000100:00080000:1.0:1788574813.148041:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574813.148047:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574813.148055:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574813.148076:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569885952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574813.148085:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574813.148089:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574813.148093:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574813.148097:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11566 00000100:00080000:1.0:1788574813.148101:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574813.148105:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574813.148113:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574813.148120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574813.148131:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569886080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574813.148146:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574813.148152:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574813.148155:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574813.148162:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11566 00000100:00080000:1.0:1788574813.148169:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574813.148174:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574813.148181:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574813.148186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff920a52893b80 x1875444569885696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574813.148201:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569886208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574813.148221:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569885696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.148236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574813.148242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574813.148252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569885696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574813.148287:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574813.148315:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574813.148328:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff920a52890000 x1875444569885824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574813.148340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569885824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.148355:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574813.148359:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574813.148362:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574813.148370:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574813.148389:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574813.148400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff920a52890a80 x1875444569885952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574813.148410:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569885952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.148418:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574813.148422:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574813.148425:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569885952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574813.148431:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574813.148452:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574813.148462:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff920a52892a00 x1875444569886080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574813.148473:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569886080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.148481:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574813.148485:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574813.148489:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574813.148495:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574813.148514:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574813.148524:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574813/real 1788574813] req@ffff920a52892680 x1875444569886208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574813.148535:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569886208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574813.148543:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574813.148581:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574813.148618:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574813.595788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574814.171795:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574814.619784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574815.195713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574815.643778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574816.219702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574816.667804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574817.243693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574817.691764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574818.267688:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574818.267697:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574818.267702:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574818.267713:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11571 00000100:00100000:2.0:1788574818.267715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574818.267725:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574818.267735:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574818.267773:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569886336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574818.267803:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574818.267806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574818.267808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574818.267811:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11571 00000100:00080000:1.0:1788574818.267815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574818.267819:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574818.267828:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574818.267838:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569886464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574818.267848:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574818.267851:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574818.267852:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574818.267856:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11571 00000100:00080000:1.0:1788574818.267859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574818.267863:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574818.267867:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574818.267870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569886336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574818.267879:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569886592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574818.267888:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574818.267891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574818.267893:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574818.267897:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11571 00000100:00080000:1.0:1788574818.267901:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574818.267904:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574818.267909:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574818.267920:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569886720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574818.267928:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574818.267932:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788574818.267935:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574818.267935:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574818.267941:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11571 00000100:00080000:1.0:1788574818.267945:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574818.267949:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574818.267954:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574818.267964:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569886848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574818.267974:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff920a52891500 x1875444569886336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.267999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569886336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574818.268009:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574818.268013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574818.268016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569886336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574818.268023:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574818.268043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574818.268054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff920a52891f80 x1875444569886464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.268063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569886464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574818.268070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574818.268072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574818.268075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574818.268095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574818.268109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574818.268118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff920a52891880 x1875444569886592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.268128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569886592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574818.268135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574818.268138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574818.268141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574818.268146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574818.268164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574818.268174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff920a52893480 x1875444569886720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.268184:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569886720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574818.268191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574818.268194:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574818.268196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574818.268202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574818.268218:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574818.268227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574818/real 1788574818] req@ffff920a52890380 x1875444569886848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574818.268237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569886848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574818.268245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574818.268247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574818.268250:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569886848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574818.715716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574819.291709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574819.739851:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574820.315687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574820.763743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574821.339687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574821.787749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574822.363696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574822.811782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574823.387765:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574823.387769:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.387782:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574823.387792:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.387802:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11576 00000100:00080000:1.0:1788574823.387814:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.387822:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574823.387854:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569886976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.387874:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.387877:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574823.387878:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.387881:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11576 00000100:00080000:1.0:1788574823.387884:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.387887:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574823.387896:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574823.387904:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569887104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.387912:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.387914:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574823.387915:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.387925:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11576 00000100:00100000:0.0:1788574823.387928:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569886976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788574823.387929:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.387941:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574823.387952:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574823.387962:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569887232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.387969:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.387973:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788574823.387974:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.387976:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11576 00000100:00080000:1.0:1788574823.387978:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.387980:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574823.387983:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574823.387989:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569887360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574823.388000:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574823.388002:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574823.388004:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574823.388006:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11576 00000100:00080000:1.0:1788574823.388014:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574823.388016:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574823.388018:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574823.388027:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569887488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574823.388038:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574823.388083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff920a52890e00 x1875444569886976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.388100:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569886976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574823.388111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574823.388114:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574823.388118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569886976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574823.388146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574823.388166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574823.388175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff920a52893800 x1875444569887104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.388183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569887104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574823.388189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574823.388191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574823.388193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574823.388198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574823.388210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574823.388216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff920a52892d80 x1875444569887232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.388224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569887232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574823.388232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574823.388236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574823.388239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574823.388245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574823.388268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574823.388280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff920a52892300 x1875444569887360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.388308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569887360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574823.388317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574823.388320:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574823.388324:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574823.388331:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574823.388354:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574823.388366:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574823/real 1788574823] req@ffff920a52891180 x1875444569887488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574823.388375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569887488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574823.388381:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574823.388384:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574823.388385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574823.835787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574824.411673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574824.859775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574825.435692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574825.883821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574826.459667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574826.907802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574827.483688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574827.931788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574828.507709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574828.507709:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574828.507719:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574828.507723:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574828.507733:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11581 00000100:00080000:3.0:1788574828.507741:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574828.507747:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574828.507778:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569887616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574828.507797:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574828.507800:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574828.507801:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574828.507804:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11581 00000100:00080000:3.0:1788574828.507806:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574828.507809:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574828.507817:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574828.507831:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569887744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574828.507832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569887616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574828.507844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574828.507847:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574828.507848:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574828.507851:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11581 00000100:00080000:3.0:1788574828.507854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574828.507856:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574828.507862:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574828.507874:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569887872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574828.507880:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574828.507885:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574828.507887:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574828.507889:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11581 00000100:00080000:3.0:1788574828.507892:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574828.507894:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574828.507897:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788574828.507900:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574828.507908:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569888000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574828.507915:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574828.507917:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574828.507918:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574828.507921:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11581 00000100:00080000:3.0:1788574828.507924:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574828.507927:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574828.507930:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574828.507937:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff920a529b3100 x1875444569887616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574828.507938:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569888128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574828.507976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569887616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574828.507988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574828.507992:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574828.507994:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569887616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574828.508012:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574828.508027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574828.508034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff920a529b1c00 x1875444569887744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574828.508041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569887744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574828.508045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574828.508047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574828.508049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574828.508052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574828.508061:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574828.508066:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff920a529b2a00 x1875444569887872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574828.508073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569887872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574828.508077:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574828.508079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574828.508080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569887872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574828.508084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574828.508092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574828.508097:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff920a529b3480 x1875444569888000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574828.508103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569888000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574828.508108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574828.508109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574828.508111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574828.508113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574828.508121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574828.508126:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574828/real 1788574828] req@ffff920a529b2d80 x1875444569888128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574828.508131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569888128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574828.508135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574828.508137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574828.508139:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574828.955835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574829.531668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574829.979721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574830.555742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574831.003782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574831.579693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574832.027699:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574832.603703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574833.051753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574833.627691:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574833.627695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574833.627703:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574833.627708:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574833.627717:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11587 00000100:00080000:3.0:1788574833.627728:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574833.627735:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574833.627773:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569888256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574833.627799:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574833.627802:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574833.627804:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574833.627807:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11587 00000100:00080000:3.0:1788574833.627810:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574833.627814:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574833.627823:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574833.627835:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569888384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574833.627840:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569888256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574833.627845:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574833.627851:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574833.627853:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574833.627857:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11587 00000100:00080000:3.0:1788574833.627861:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574833.627865:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574833.627870:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574833.627882:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569888512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574833.627890:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574833.627894:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574833.627896:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574833.627899:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11587 00000100:00080000:3.0:1788574833.627904:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788574833.627908:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574833.627911:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574833.627917:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574833.627927:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569888640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574833.627934:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574833.627936:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574833.627938:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574833.627941:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11587 00000100:00080000:3.0:1788574833.627944:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574833.627948:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574833.627949:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574833/real 1788574833] req@ffff920a529b2d80 x1875444569888256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574833.627952:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574833.627962:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569888768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574833.627976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569888256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574833.627987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574833.627990:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574833.627994:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569888256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574833.628016:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574833.628069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574833.628082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574833/real 1788574833] req@ffff920a529b3480 x1875444569888384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574833.628093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569888384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574833.628101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574833.628103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574833.628107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574833.628113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574833.628130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574833.628140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574833/real 1788574833] req@ffff920a529b2a00 x1875444569888512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574833.628151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569888512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574833.628162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574833.628166:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574833.628168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574833.628172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574833.628189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574833.628198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574833/real 1788574833] req@ffff920a529b1c00 x1875444569888640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574833.628208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569888640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574833.628214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574833.628217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574833.628220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574833.628224:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574833.628239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574833.628247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574833/real 1788574833] req@ffff920a529b3100 x1875444569888768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574833.628258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569888768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574833.628265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574833.628268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574833.628272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569888768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574834.075791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574834.651717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574835.100723:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574835.675704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574836.123769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574836.699717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574837.147772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574837.723677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574838.171786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574838.747671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574838.747715:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574838.747724:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574838.747730:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574838.747742:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11592 00000100:00080000:0.0:1788574838.747753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574838.747761:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574838.747802:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569888896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574838.747825:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574838.747828:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574838.747830:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574838.747833:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11592 00000100:00080000:0.0:1788574838.747836:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574838.747839:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574838.747850:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574838.747858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569888896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574838.747861:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569889024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574838.747874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574838.747877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574838.747878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574838.747881:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11592 00000100:00080000:0.0:1788574838.747883:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574838.747886:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574838.747890:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574838.747898:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569889152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574838.747921:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574838.747924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574838.747926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574838.747930:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11592 00000100:00080000:0.0:1788574838.747940:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574838.747944:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574838.747948:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574838.747960:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569889280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574838.747961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574838.747970:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574838.747973:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574838.747975:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574838.747978:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11592 00000100:00080000:0.0:1788574838.747981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574838.747984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574838.747987:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574838.747995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569889408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574838.748005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574838/real 1788574838] req@ffff920a48350700 x1875444569888896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574838.748023:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569888896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574838.748032:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574838.748036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574838.748039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569888896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574838.748061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574838.748079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574838.748087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574838/real 1788574838] req@ffff920a48350a80 x1875444569889024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574838.748096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569889024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574838.748101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574838.748103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574838.748105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574838.748109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574838.748121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574838.748127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574838/real 1788574838] req@ffff920a48353480 x1875444569889152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574838.748134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569889152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574838.748144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574838.748146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574838.748149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574838.748152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574838.748162:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574838.748168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574838/real 1788574838] req@ffff920a48353800 x1875444569889280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574838.748175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569889280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574838.748180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574838.748182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574838.748184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574838.748187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574838.748197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574838.748203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574838/real 1788574838] req@ffff920a48351c00 x1875444569889408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574838.748209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569889408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574838.748214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574838.748216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574838.748220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574839.195758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574839.771708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574840.219750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574840.795696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574841.243757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574841.819687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574842.267777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574842.843692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574843.291840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574843.867669:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574843.867670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574843.867685:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574843.867689:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574843.867699:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11597 00000100:00080000:0.0:1788574843.867710:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574843.867718:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574843.867756:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569889536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574843.867813:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574843.867817:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574843.867820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574843.867826:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11597 00000100:00100000:1.0:1788574843.867830:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569889536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574843.867831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574843.867838:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574843.867850:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574843.867873:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569889664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574843.867884:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574843.867888:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574843.867890:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574843.867894:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11597 00000100:00080000:0.0:1788574843.867898:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574843.867904:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574843.867907:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574843.867920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574843.867934:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569889792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574843.867946:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574843.867950:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574843/real 1788574843] req@ffff920a48351500 x1875444569889536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574843.867952:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574843.867956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574843.867965:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11597 00000100:00080000:0.0:1788574843.867970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574843.867974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574843.867978:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569889536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788574843.867981:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574843.867995:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574843.868001:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569889920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574843.868002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574843.868011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569889536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574843.868019:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574843.868024:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574843.868027:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574843.868031:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11597 00000100:00080000:0.0:1788574843.868035:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574843.868036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574843.868044:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574843.868050:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574843.868064:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569890048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574843.868065:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574843.868076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574843/real 1788574843] req@ffff920a48353b80 x1875444569889664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574843.868097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569889664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574843.868106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574843.868110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574843.868114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574843.868121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574843.868149:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574843.868160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574843/real 1788574843] req@ffff920a48350380 x1875444569889792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574843.868172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569889792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574843.868180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574843.868183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574843.868187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574843.868193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574843.868215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574843.868225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574843/real 1788574843] req@ffff920a48351c00 x1875444569889920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574843.868235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569889920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574843.868251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574843.868254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574843.868257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569889920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574843.868273:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574843.868298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574843.868311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574843/real 1788574843] req@ffff920a48353800 x1875444569890048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574843.868324:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569890048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574843.868332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574843.868335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574843.868338:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574844.315736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574844.891697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574845.339785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574845.915665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574846.363872:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574846.939723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574847.387822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574847.963728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574848.411826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574848.987708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574848.987717:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574848.987728:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574848.987735:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574848.987748:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11602 00000100:00080000:3.0:1788574848.987761:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574848.987771:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574848.987822:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569890176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574848.987856:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574848.987860:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574848.987862:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574848.987868:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11602 00000100:00080000:3.0:1788574848.987873:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574848.987879:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574848.987892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574848.987896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569890176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574848.987910:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569890304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574848.987923:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574848.987928:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574848.987930:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574848.987935:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11602 00000100:00080000:3.0:1788574848.987940:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574848.987945:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574848.987951:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574848.987964:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569890432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574848.987975:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574848.987979:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788574848.987983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574848.987984:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574848.987992:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11602 00000100:00080000:3.0:1788574848.987996:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574848.988002:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574848.988008:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574848.988022:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569890560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574848.988024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574848/real 1788574848] req@ffff920a529b1f80 x1875444569890176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574848.988055:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574848.988061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569890176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574848.988061:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574848.988065:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574848.988077:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11602 00000100:00080000:1.0:1788574848.988081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574848.988082:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574848.988087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574848.988091:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574848.988095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569890176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574848.988101:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574848.988115:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569890688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574848.988122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574848.988147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574848.988158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574848/real 1788574848] req@ffff920a529b0000 x1875444569890304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574848.988167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569890304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574848.988174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574848.988176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574848.988179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574848.988182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574848.988193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574848.988199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574848/real 1788574848] req@ffff920a529b0700 x1875444569890432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574848.988207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569890432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574848.988212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574848.988214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574848.988216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574848.988219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574848.988229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574848.988235:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574848/real 1788574848] req@ffff920a529b2300 x1875444569890560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574848.988242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569890560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574848.988250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574848.988253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574848.988256:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574848.988272:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574848.988292:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574848.988303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574848/real 1788574848] req@ffff920a529b0380 x1875444569890688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574848.988313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569890688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574873 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574848.988319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574848.988321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574848.988323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574849.435716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574850.011704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574850.459799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574851.035697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574851.483734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574852.059700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574852.507802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574853.083674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574853.531833:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574854.107724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574854.107732:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574854.107743:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574854.107748:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574854.107764:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11607 00000100:00080000:3.0:1788574854.107777:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574854.107787:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574854.107838:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569890816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574854.107870:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574854.107875:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574854.107877:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574854.107881:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11607 00000100:00080000:3.0:1788574854.107885:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574854.107891:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574854.107904:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574854.107924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569890944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574854.107929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569890816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574854.107942:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574854.107948:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574854.107950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574854.107955:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11607 00000100:00080000:3.0:1788574854.107960:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574854.107967:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574854.107973:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574854.107989:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569891072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574854.108000:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574854.108004:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574854.108007:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574854.108013:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11607 00000100:00080000:3.0:1788574854.108018:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574854.108023:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788574854.108030:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574854.108030:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574854.108049:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569891200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574854.108060:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574854.108064:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574854.108066:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574854.108072:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11607 00000100:00080000:3.0:1788574854.108076:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574854.108079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff920a529b3b80 x1875444569890816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574854.108085:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574854.108098:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574854.108114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569890816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574854.108117:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569891328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574854.108189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574854.108199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574854.108205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569890816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574854.108235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574854.108263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574854.108277:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff920a529b0e00 x1875444569890944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574854.108289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569890944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574854.108298:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574854.108301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574854.108304:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569890944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574854.108311:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574854.108330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574854.108341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff920a529b1180 x1875444569891072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574854.108351:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569891072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574854.108359:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574854.108363:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574854.108366:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574854.108372:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574854.108393:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574854.108403:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff920a529b1500 x1875444569891200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574854.108414:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569891200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574854.108422:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574854.108426:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574854.108429:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574854.108435:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574854.108456:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574854.108469:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574854/real 1788574854] req@ffff920a529b0380 x1875444569891328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574854.108480:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569891328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574854.108487:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574854.108490:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574854.108493:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574854.555841:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574855.131763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574855.579847:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574856.155726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574856.603806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574857.179727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574857.627788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574858.203706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574858.651802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574859.227736:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574859.227746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574859.227760:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574859.227765:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.227777:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11612 00000100:00080000:0.0:1788574859.227789:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.227796:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574859.227856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569891456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.227886:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.227891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574859.227893:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.227899:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11612 00000100:00080000:0.0:1788574859.227904:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.227910:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574859.227920:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569891456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788574859.227924:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574859.227943:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569891584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.227956:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.227960:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574859.227963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.227968:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11612 00000100:00080000:0.0:1788574859.227981:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.227985:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574859.227990:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788574859.227997:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574859.228002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569891712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574859.228011:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.228014:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574859.228016:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.228019:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11612 00000100:00080000:0.0:1788574859.228022:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.228025:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574859.228028:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574859.228036:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569891840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574859.228041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574859] req@ffff920a48353480 x1875444569891456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574859.228047:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574859.228056:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574859.228059:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574859.228068:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11612 00000100:00080000:0.0:1788574859.228072:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574859.228076:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574859.228082:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574859.228083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569891456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574859.228100:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569891968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574859.228101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574859.228109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574859.228117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569891456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574859.228145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574859.228165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574859.228174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574859] req@ffff920a48350a80 x1875444569891584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574859.228183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569891584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574859.228189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574859.228192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574859.228194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574859.228198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574859.228209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574859.228214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574859] req@ffff920a48350700 x1875444569891712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574859.228222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569891712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574859.228228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574859.228230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574859.228231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574859.228235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574859.228246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574859.228251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574859] req@ffff920a48353800 x1875444569891840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574859.228259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569891840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574859.228263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574859.228265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574859.228267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574859.228270:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574859.228280:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574859.228285:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574859/real 1788574859] req@ffff920a48351c00 x1875444569891968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574859.228292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569891968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574859.228302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574859.228304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574859.228306:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569891968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574859.675736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574860.251701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574860.699755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574861.275714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574861.723736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574862.299739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574862.747743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574863.323703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574863.771775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574864.347706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574864.347708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574864.347718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574864.347722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574864.347733:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11617 00000100:00080000:3.0:1788574864.347746:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574864.347755:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574864.347802:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569892096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574864.347833:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574864.347837:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574864.347840:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574864.347845:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11617 00000100:00080000:3.0:1788574864.347858:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574864.347865:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574864.347871:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569892096:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574864.347881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574864.347899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569892224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574864.347913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574864.347917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574864.347919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574864.347925:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11617 00000100:00080000:3.0:1788574864.347930:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574864.347935:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574864.347942:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574864.347961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569892352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574864.347974:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574864.347977:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574864.347980:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788574864.347983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574864.347987:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11617 00000100:00080000:3.0:1788574864.347992:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574864.347998:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574864.348006:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574864.348021:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569892480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574864.348033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff920a529b2300 x1875444569892096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.348058:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574864.348062:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574864.348063:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574864.348068:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11617 00000100:00080000:3.0:1788574864.348072:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788574864.348075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569892096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574864.348078:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574864.348086:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574864.348098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788574864.348102:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569892608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574864.348108:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574864.348119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569892096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574864.348150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574864.348179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574864.348193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff920a529b0700 x1875444569892224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574864.348205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569892224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574864.348214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574864.348217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574864.348221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574864.348227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574864.348246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574864.348259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff920a529b0000 x1875444569892352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574864.348309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569892352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574864.348318:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574864.348322:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574864.348326:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574864.348332:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574864.348353:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574864.348363:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff920a529b1f80 x1875444569892480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574864.348374:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569892480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574864.348381:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574864.348385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574864.348388:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574864.348394:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574864.348411:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574864.348422:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574864/real 1788574864] req@ffff920a529b3100 x1875444569892608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574864.348433:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569892608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574864.348441:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574864.348444:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574864.348448:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574864.795787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574865.371720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574865.819756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574866.395715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574866.843749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574867.419696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574867.867810:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574868.443677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574868.891754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574869.467700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574869.467822:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574869.467829:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574869.467834:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574869.467846:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11622 00000100:00080000:2.0:1788574869.467859:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574869.467868:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574869.467919:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569892736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574869.467953:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574869.467956:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574869.467958:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574869.467961:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11622 00000100:00080000:2.0:1788574869.467965:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574869.467968:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574869.467985:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574869.467997:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569892864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574869.468007:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569892736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788574869.468008:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574869.468015:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574869.468018:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574869.468022:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11622 00000100:00080000:2.0:1788574869.468025:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574869.468027:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574869.468030:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574869.468041:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569892992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574869.468049:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574869.468052:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574869.468053:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574869.468056:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11622 00000100:00080000:2.0:1788574869.468059:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574869.468062:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574869.468066:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574869.468073:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569893120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574869.468079:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574869.468082:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574869.468084:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574869.468086:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11622 00000100:00080000:2.0:1788574869.468091:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574869.468093:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574869.468103:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574869.468109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574869.468115:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569893248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574869.468177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff920a60ac7800 x1875444569892736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.468201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569892736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.468211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.468214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574869.468218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569892736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574869.468247:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574869.468280:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574869.468291:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff920a60ac4e00 x1875444569892864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.468316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569892864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.468326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.468330:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574869.468333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574869.468340:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574869.468363:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574869.468376:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff920a60ac4380 x1875444569892992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.468387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569892992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.468393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.468395:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574869.468397:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569892992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574869.468402:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574869.468419:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574869.468431:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff920a60ac5c00 x1875444569893120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.468444:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569893120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.468453:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.468457:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574869.468461:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574869.468467:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574869.468486:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574869.468495:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574869/real 1788574869] req@ffff920a60ac4000 x1875444569893248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574869.468503:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569893248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574869.468509:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574869.468511:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574869.468513:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574869.915735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574870.491734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574870.939710:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574871.515723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574871.963806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574872.539743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574872.987752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574873.563708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574874.011803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574874.587707:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574874.587707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574874.587717:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574874.587721:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574874.587731:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11627 00000100:00080000:0.0:1788574874.587742:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574874.587751:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574874.587789:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569893376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574874.587813:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574874.587816:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574874.587817:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574874.587820:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11627 00000100:00080000:0.0:1788574874.587823:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574874.587825:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574874.587834:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574874.587844:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569893504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574874.587853:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574874.587855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569893376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574874.587856:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574874.587860:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574874.587866:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11627 00000100:00080000:0.0:1788574874.587868:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574874.587870:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574874.587873:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574874.587882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569893632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574874.587888:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574874.587891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574874.587893:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574874.587895:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11627 00000100:00080000:0.0:1788574874.587899:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574874.587902:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574874.587906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574874.587916:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569893760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574874.587922:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574874.587925:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574874.587927:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574874.587929:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11627 00000100:00080000:0.0:1788574874.587931:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574874.587933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574874.587936:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574874.587944:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569893888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574874.587960:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574874.587986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574874.588002:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574874.588034:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574875.035765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574875.611687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574876.059754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574876.635683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574877.083746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574877.659701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574877.659842:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574877.659883:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574877.659898:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574877.659951:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574877.659962:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574877.659968:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574877.659973:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788574877.660173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574877] req@ffff920a48353b80 x1875444569893504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574877.660197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569893504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574877.660209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574877.660214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574877.660218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574877.660225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574877] req@ffff920a48351500 x1875444569893632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574877.660236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569893632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574877.660243:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574877.660246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574877.660249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574877.660254:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574877] req@ffff920a48351c00 x1875444569893760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574877.660263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569893760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574877.660269:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574877.660272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574877.660275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574877.660280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574877] req@ffff920a48353800 x1875444569893888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574877.660289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569893888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574877.660296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574877.660299:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574877.660301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569893888:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574877.660307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574874/real 1788574877] req@ffff920a48350380 x1875444569893376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574877.660318:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569893376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574877.660325:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574877.660327:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574877.660330:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569893376:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788574878.107761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574878.683681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574879.131737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574879.707708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574879.707713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574879.707722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574879.707728:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574879.707737:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11633 00000100:00080000:0.0:1788574879.707747:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574879.707754:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574879.707790:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569894016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574879.707815:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574879.707819:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574879.707821:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574879.707825:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11633 00000100:00080000:0.0:1788574879.707829:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574879.707833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574879.707843:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574879.707848:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569894016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574879.707860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569894144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574879.707873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574879.707877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574879.707879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574879.707883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11633 00000100:00080000:0.0:1788574879.707888:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574879.707892:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574879.707897:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574879.707929:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569894272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574879.707938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574879.707940:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574879.707944:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574879.707946:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574879.707950:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11633 00000100:00080000:0.0:1788574879.707954:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574879.707957:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574879.707962:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574879.707974:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569894400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574879.707982:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574879/real 1788574879] req@ffff920a48350700 x1875444569894016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574879.707986:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574879.707995:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574879.707999:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574879.708007:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11633 00000100:00080000:0.0:1788574879.708011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574879.708015:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569894016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574879.708016:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574879.708022:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574879.708033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788574879.708036:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569894528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574879.708041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574879.708049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569894016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574879.708079:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574879.708103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574879.708117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574879/real 1788574879] req@ffff920a48350a80 x1875444569894144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574879.708129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569894144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574879.708137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574879.708140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574879.708144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574879.708150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574879.708169:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574879.708179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574879/real 1788574879] req@ffff920a48353480 x1875444569894272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574879.708192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569894272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574879.708206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574879.708209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574879.708212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574879.708218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574879.708236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574879.708244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574879/real 1788574879] req@ffff920a48350380 x1875444569894400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574879.708255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569894400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574879.708262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574879.708265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574879.708268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574879.708274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574879.708293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574879.708303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574879/real 1788574879] req@ffff920a48353800 x1875444569894528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574879.708314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569894528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574879.708321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574879.708325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574879.708328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574880.155746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574880.731669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574881.179752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574881.755688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574882.203737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574882.779674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574883.227711:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574883.803687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574884.251719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574884.827695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574884.827702:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574884.827710:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574884.827714:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574884.827724:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11638 00000100:00080000:0.0:1788574884.827736:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574884.827744:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574884.827782:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569894656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574884.827803:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574884.827806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574884.827808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574884.827811:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11638 00000100:00080000:0.0:1788574884.827814:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574884.827819:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574884.827827:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574884.827836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569894656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574884.827841:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569894784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574884.827851:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574884.827854:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574884.827855:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574884.827858:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11638 00000100:00080000:0.0:1788574884.827860:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574884.827862:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574884.827865:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574884.827889:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569894912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574884.827896:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574884.827899:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574884.827901:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574884.827904:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11638 00000100:00080000:0.0:1788574884.827907:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574884.827912:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574884.827915:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574884.827923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574884.827939:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569895040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574884.827947:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574884/real 1788574884] req@ffff920a48351c00 x1875444569894656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574884.827976:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574884.827978:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569894656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574884.827980:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574884.827984:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574884.827993:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11638 00000100:00080000:0.0:1788574884.827999:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574884.827999:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574884.828005:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574884.828010:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788574884.828014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574884.828019:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569894656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574884.828039:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569895168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574884.828044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574884.828081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574884.828096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574884/real 1788574884] req@ffff920a48351500 x1875444569894784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574884.828106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569894784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574884.828113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574884.828115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574884.828118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574884.828122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574884.828137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574884.828143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574884/real 1788574884] req@ffff920a48353b80 x1875444569894912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574884.828151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569894912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574884.828156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574884.828158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574884.828160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569894912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574884.828164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574884.828175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574884.828181:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574884/real 1788574884] req@ffff920a48353800 x1875444569895040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574884.828189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569895040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574884.828194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574884.828196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574884.828198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574884.828210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574884.828221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574884.828227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574884/real 1788574884] req@ffff920a48350380 x1875444569895168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574884.828234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569895168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574884.828240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574884.828242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574884.828245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574885.275743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574885.851685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574886.299743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574886.875699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574887.323809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574887.899677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574888.347727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574888.923693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574889.371770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574889.947667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788574889.947685:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574889.947692:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788574889.947697:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574889.947707:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11643 00000100:00080000:2.0:1788574889.947718:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574889.947726:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574889.947759:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569895296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574889.947784:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574889.947789:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788574889.947791:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574889.947796:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11643 00000100:00080000:2.0:1788574889.947802:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574889.947809:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574889.947819:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569895296:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788574889.947820:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574889.947865:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569895424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574889.947875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574889.947878:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788574889.947880:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574889.947884:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11643 00000100:00080000:2.0:1788574889.947887:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574889.947890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574889.947895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574889.947903:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569895552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788574889.947911:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788574889.947914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574889.947918:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788574889.947919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574889.947923:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11643 00000100:00080000:2.0:1788574889.947927:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574889.947932:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788574889.947938:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574889.947951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574889/real 1788574889] req@ffff920a60ac5500 x1875444569895296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574889.947958:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569895680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574889.947976:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574889.947981:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788574889.947982:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569895296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574889.947983:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574889.947990:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11643 00000100:00080000:2.0:1788574889.947997:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574889.948003:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788574889.948005:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788574889.948008:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788574889.948013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574889.948021:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569895808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574889.948025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569895296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574889.948052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574889.948072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574889.948082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574889/real 1788574889] req@ffff920a60ac6300 x1875444569895424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574889.948090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569895424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574889.948096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574889.948098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574889.948100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574889.948104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574889.948115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574889.948121:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574889/real 1788574889] req@ffff920a60ac4700 x1875444569895552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574889.948134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569895552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574889.948139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574889.948141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574889.948143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574889.948147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574889.948158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574889.948164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574889/real 1788574889] req@ffff920a60ac4a80 x1875444569895680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574889.948172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569895680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574889.948176:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574889.948178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574889.948180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574889.948194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574889.948213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574889.948224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574889/real 1788574889] req@ffff920a60ac6a00 x1875444569895808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574889.948237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569895808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574914 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574889.948251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574889.948254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574889.948256:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569895808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574890.395790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574890.971695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574891.419815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574891.995736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574892.443840:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574893.019727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574893.467846:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574894.043733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574894.491849:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574895.067712:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574895.067712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574895.067722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574895.067726:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.067734:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11648 00000100:00080000:0.0:1788574895.067744:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.067751:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574895.067786:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569895936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574895.067823:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.067826:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574895.067827:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.067831:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11648 00000100:00080000:0.0:1788574895.067838:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.067841:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574895.067851:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574895.067862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569896064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574895.067870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569895936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574895.067873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.067878:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574895.067880:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.067883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11648 00000100:00080000:0.0:1788574895.067886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.067888:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574895.067891:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574895.067899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569896192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574895.067906:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.067909:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574895.067910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.067913:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11648 00000100:00080000:0.0:1788574895.067917:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.067919:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574895.067923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574895.067930:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574895.067933:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569896320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574895.067943:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574895.067946:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574895.067949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574895.067953:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11648 00000100:00080000:0.0:1788574895.067956:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574895.067961:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574895.067967:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574895.067971:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff920a48353480 x1875444569895936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574895.067984:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569896448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574895.067990:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569895936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574895.068012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574895.068018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574895.068023:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569895936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574895.068048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574895.068074:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574895.068089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff920a48350a80 x1875444569896064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574895.068104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569896064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574895.068114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574895.068118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574895.068122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574895.068129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574895.068151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574895.068164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff920a48350700 x1875444569896192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574895.068178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569896192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574895.068187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574895.068197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574895.068200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574895.068205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574895.068222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574895.068230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff920a48350380 x1875444569896320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574895.068238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569896320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574895.068248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574895.068250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574895.068252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574895.068256:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574895.068266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574895.068272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574895/real 1788574895] req@ffff920a48353800 x1875444569896448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574895.068279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569896448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574895.068284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574895.068286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574895.068288:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574895.515745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574896.091689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574896.539733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574897.115698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574897.563744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574898.139695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574898.587731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574899.163692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574899.611866:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574900.187676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574900.187686:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574900.187693:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574900.187698:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574900.187709:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11653 00000100:00080000:0.0:1788574900.187722:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574900.187731:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574900.187787:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569896576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574900.187817:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574900.187821:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574900.187823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574900.187828:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11653 00000100:00080000:0.0:1788574900.187832:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574900.187837:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574900.187847:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574900.187865:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569896704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574900.187884:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574900.187888:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574900.187890:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574900.187900:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11653 00000100:00100000:1.0:1788574900.187901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569896576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574900.187905:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574900.187916:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574900.187923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574900.187937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569896832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574900.187947:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574900.187951:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574900.187953:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574900.187958:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11653 00000100:00080000:0.0:1788574900.187962:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574900.187967:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574900.187974:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574900.187987:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569896960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574900.188000:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574900.188003:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574900.188005:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574900.188009:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11653 00000100:00080000:0.0:1788574900.188015:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788574900.188022:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788574900.188028:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574900.188035:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574900.188049:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569897088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574900.188085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff920a48353b80 x1875444569896576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.188111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569896576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.188126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.188132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574900.188137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569896576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574900.188173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574900.188200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574900.188215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff920a48351500 x1875444569896704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.188229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569896704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.188238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.188242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574900.188245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574900.188253:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574900.188276:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574900.188289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff920a48351c00 x1875444569896832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.188301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569896832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.188311:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.188315:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574900.188318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574900.188326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574900.188349:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574900.188362:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff920a48353800 x1875444569896960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.188374:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569896960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.188383:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.188386:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574900.188390:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569896960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574900.188397:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574900.188421:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574900.188434:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574900/real 1788574900] req@ffff920a48350380 x1875444569897088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574900.188446:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569897088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574900.188454:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574900.188457:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574900.188460:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574900.635716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574901.211803:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574901.659823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574902.235699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574902.683798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574903.259712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574903.707775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574904.283755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574904.731822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574905.307709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574905.307728:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.307735:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574905.307741:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.307756:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11658 00000100:00080000:0.0:1788574905.307772:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.307780:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574905.307827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569897216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.307849:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.307852:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.307854:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.307857:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11658 00000100:00080000:0.0:1788574905.307860:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.307863:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574905.307873:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.307883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569897344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574905.307891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569897216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574905.307892:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.307900:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574905.307904:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.307910:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11658 00000100:00080000:0.0:1788574905.307913:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.307916:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574905.307922:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574905.307934:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569897472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.307946:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.307950:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574905.307952:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.307957:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11658 00000100:00080000:0.0:1788574905.307963:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.307966:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574905.307972:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574905.307983:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574905.307987:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569897600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574905.307995:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574905.307998:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574905.307999:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574905.308001:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11658 00000100:00080000:0.0:1788574905.308004:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574905.308006:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574905.308011:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574905.308018:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569897728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574905.308030:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff920a48350700 x1875444569897216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.308049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569897216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.308059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.308063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574905.308067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569897216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574905.308091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574905.308110:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574905.308119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff920a48350a80 x1875444569897344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.308127:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569897344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.308133:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.308135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574905.308137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574905.308141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574905.308153:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574905.308159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff920a48353480 x1875444569897472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.308166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569897472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.308172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.308174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574905.308176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574905.308180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574905.308191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574905.308196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff920a48350380 x1875444569897600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.308203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569897600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.308208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.308210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574905.308212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574905.308215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574905.308225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574905.308231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574905/real 1788574905] req@ffff920a48353800 x1875444569897728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574905.308239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569897728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574905.308244:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574905.308245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574905.308247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574905.755753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574906.331784:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574906.779788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574907.355683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574907.803770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574908.379740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574908.827764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574909.403700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574909.851780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788574910.427790:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574910.427798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574910.427799:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574910.427805:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.427815:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11663 00000100:00080000:0.0:1788574910.427825:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.427833:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574910.427869:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569897856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.427890:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.427893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.427894:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.427897:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11663 00000100:00080000:0.0:1788574910.427899:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.427903:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574910.427911:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.427921:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569897984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574910.427927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569897856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788574910.427944:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.427947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574910.427948:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.427952:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11663 00000100:00080000:0.0:1788574910.427957:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.427961:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574910.427964:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574910.427974:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569898112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.427981:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.427984:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574910.427986:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.427989:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11663 00000100:00080000:0.0:1788574910.428001:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.428004:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574910.428008:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788574910.428020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574910.428021:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569898240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574910.428033:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574910.428035:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574910.428036:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574910.428039:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11663 00000100:00080000:0.0:1788574910.428042:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574910.428044:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574910.428049:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574910.428057:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569898368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574910.428091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff920a48351c00 x1875444569897856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.428112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569897856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.428125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.428130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574910.428134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569897856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574910.428163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574910.428183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574910.428192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff920a48351500 x1875444569897984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.428201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569897984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.428206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.428208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574910.428210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569897984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574910.428215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788574910.428226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574910.428232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff920a48353b80 x1875444569898112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.428239:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569898112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.428244:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.428246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574910.428248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574910.428252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574910.428263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574910.428268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff920a48353800 x1875444569898240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.428283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569898240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.428289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.428291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574910.428293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788574910.428297:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788574910.428309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788574910.428321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574910/real 1788574910] req@ffff920a48350380 x1875444569898368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788574910.428331:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569898368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574910.428336:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788574910.428338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788574910.428340:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574910.875815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574911.451705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574911.899803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574912.475734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574912.923757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574913.499728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574913.947768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574914.523708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574914.971824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574915.547729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574915.547799:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574915.547805:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788574915.547810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574915.547823:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11668 00000100:00080000:1.0:1788574915.547835:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574915.547843:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574915.547882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569898496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574915.547913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574915.547918:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574915.547920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574915.547927:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11668 00000100:00080000:1.0:1788574915.547932:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574915.547939:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574915.547950:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574915.547958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569898496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788574915.547970:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569898624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574915.547987:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574915.547992:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574915.547995:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574915.548000:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11668 00000100:00080000:1.0:1788574915.548006:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574915.548012:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574915.548021:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574915.548035:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569898752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574915.548046:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574915.548052:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1788574915.548053:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574915.548058:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574915.548068:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11668 00000100:00080000:1.0:1788574915.548075:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574915.548081:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574915.548089:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574915.548114:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569898880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574915.548126:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574915.548129:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574915.548131:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574915.548135:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11668 00000100:00080000:1.0:1788574915.548139:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574915.548144:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574915.548150:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574915.548162:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569899008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574915.548165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff920a52890700 x1875444569898496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574915.548208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569898496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574915.548222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574915.548228:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574915.548233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569898496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574915.548265:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574915.548294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574915.548471:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff920a52893100 x1875444569898624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574915.548535:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569898624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574915.548552:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574915.548558:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574915.548563:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574915.548576:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788574915.548670:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574915.548694:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff920a52891180 x1875444569898752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574915.548715:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569898752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574915.548725:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574915.548729:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574915.548732:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574915.548740:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574915.548764:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574915.548777:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff920a52892300 x1875444569898880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574915.548790:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569898880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574915.548798:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574915.548802:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574915.548806:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569898880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574915.548813:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788574915.548834:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574915.548847:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574915/real 1788574915] req@ffff920a52892d80 x1875444569899008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574915.548861:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569899008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574915.548878:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574915.548882:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574915.548885:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574915.995819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574916.571718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574917.019763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574917.595699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574918.043804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574918.619727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574919.067779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574919.643687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574920.091733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788574920.667713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574920.667722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1788574920.667723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788574920.667726:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574920.667736:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11674 00000100:00080000:1.0:1788574920.667747:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574920.667754:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788574920.667788:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569899136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574920.667817:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574920.667822:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788574920.667825:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574920.667830:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11674 00000100:00080000:1.0:1788574920.667836:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574920.667841:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574920.667852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569899136:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574920.667853:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574920.667870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569899264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574920.667878:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574920.667881:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788574920.667883:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574920.667887:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11674 00000100:00080000:1.0:1788574920.667890:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574920.667893:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574920.667898:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788574920.667907:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569899392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574920.667913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788574920.667917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788574920.667919:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788574920.667919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788574920.667927:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11674 00000100:00080000:1.0:1788574920.667930:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788574920.667933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788574920.667937:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788574920.667945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569899520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788574920.667952:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574920.667957:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574920/real 1788574920] req@ffff920a52893800 x1875444569899136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574920.667958:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788574920.667963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788574920.667978:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569899136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788574920.667978:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11674 00000100:00080000:1.0:1788574920.667986:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574920.667991:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574920.667997:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788574920.668001:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574920.668004:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569899136:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788574920.668011:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574920.668030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788574920.668031:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569899648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574920.668053:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574920.668065:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574920/real 1788574920] req@ffff920a52890e00 x1875444569899264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574920.668073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569899264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574920.668079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574920.668081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574920.668083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574920.668089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574920.668108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574920.668120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574920/real 1788574920] req@ffff920a52890380 x1875444569899392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574920.668133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569899392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574920.668143:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574920.668145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574920.668147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574920.668151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574920.668174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574920.668182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574920/real 1788574920] req@ffff920a52893480 x1875444569899520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574920.668189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569899520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574920.668194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574920.668196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574920.668198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574920.668209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574920.668219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574920.668225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574920/real 1788574920] req@ffff920a52891880 x1875444569899648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574920.668232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569899648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574920.668238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574920.668240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574920.668241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574921.115768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574921.691684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574922.139784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574922.715748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574923.163800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574923.739731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574924.187777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574924.763688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574925.211758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574925.787750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574925.787756:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574925.787762:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574925.787766:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574925.787776:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11679 00000100:00080000:3.0:1788574925.787785:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574925.787792:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574925.787826:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569899776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574925.787848:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574925.787851:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574925.787852:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574925.787856:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11679 00000100:00080000:3.0:1788574925.787859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574925.787862:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574925.787871:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574925.787880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569899904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574925.787891:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574925.787896:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:0.0:1788574925.787898:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569899776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574925.787900:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574925.787914:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11679 00000100:00080000:3.0:1788574925.787917:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574925.787919:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574925.787923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574925.787934:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569900032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574925.787941:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574925.787943:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574925.787944:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574925.787948:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11679 00000100:00080000:3.0:1788574925.787949:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574925.787952:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574925.787955:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574925.787961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569900160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574925.787967:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574925.787969:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574925.787972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574925.787975:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11679 00000100:00080000:3.0:1788574925.787977:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574925.787980:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574925.787982:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574925.787993:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569900288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574925.788010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574925.788062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574925/real 1788574925] req@ffff920a529b1c00 x1875444569899776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574925.788086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569899776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574925.788099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574925.788104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574925.788109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569899776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574925.788153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574925.788181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574925.788196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574925/real 1788574925] req@ffff920a529b2a00 x1875444569899904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574925.788210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569899904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574925.788230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574925.788234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574925.788238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569899904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574925.788247:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574925.788282:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574925.788297:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574925/real 1788574925] req@ffff920a529b3480 x1875444569900032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574925.788310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569900032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574925.788319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574925.788323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574925.788327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574925.788334:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574925.788359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574925.788372:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574925/real 1788574925] req@ffff920a529b2d80 x1875444569900160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574925.788385:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569900160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574925.788395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574925.788399:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574925.788403:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574925.788410:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574925.788432:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574925.788445:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574925/real 1788574925] req@ffff920a529b3100 x1875444569900288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574925.788458:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569900288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574925.788467:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574925.788471:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574925.788474:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574926.235794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574926.811669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574927.259704:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574927.835706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574928.283771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574928.859675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574929.307758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574929.883700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574930.331749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574930.907697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574930.907731:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574930.907741:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574930.907748:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574930.907762:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11684 00000100:00080000:3.0:1788574930.907776:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574930.907786:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574930.907831:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569900416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574930.907857:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574930.907860:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574930.907861:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574930.907864:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11684 00000100:00080000:3.0:1788574930.907868:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574930.907871:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574930.907882:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574930.907900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569900416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574930.907901:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569900544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574930.907927:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574930.907932:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574930.907935:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574930.907942:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11684 00000100:00080000:3.0:1788574930.907947:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574930.907954:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574930.907961:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788574930.907975:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574930.907982:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569900672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574930.907996:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574930.908001:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574930.908003:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574930.908011:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11684 00000100:00100000:0.0:1788574930.908013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574930/real 1788574930] req@ffff920a529b1f80 x1875444569900416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574930.908021:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574930.908033:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574930.908037:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569900416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788574930.908049:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788574930.908050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574930.908058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574930.908064:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569900416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574930.908066:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569900800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574930.908082:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574930.908085:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574930.908088:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574930.908092:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11684 00000100:00080000:3.0:1788574930.908095:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788574930.908098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574930.908105:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574930.908111:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574930.908129:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569900928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574930.908136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574930.908149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574930/real 1788574930] req@ffff920a529b0000 x1875444569900544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574930.908163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569900544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574930.908193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574930.908198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574930.908202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574930.908210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574930.908234:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574930.908262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574930/real 1788574930] req@ffff920a529b0700 x1875444569900672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574930.908274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569900672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574930.908280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574930.908282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574930.908284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574930.908289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574930.908306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574930.908313:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574930/real 1788574930] req@ffff920a529b2300 x1875444569900800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574930.908320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569900800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574930.908325:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574930.908326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574930.908329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574930.908342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574930.908358:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574930.908368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574930/real 1788574930] req@ffff920a529b0380 x1875444569900928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574930.908375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569900928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574930.908380:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574930.908382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574930.908384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569900928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574931.355761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574931.931710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574932.379786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574932.955741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574933.403797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574933.979744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574934.427792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574935.003712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574935.451779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574936.027752:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574936.027761:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574936.027761:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574936.027773:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574936.027785:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11689 00000100:00080000:3.0:1788574936.027795:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574936.027801:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574936.027834:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569901056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574936.027860:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574936.027862:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574936.027864:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574936.027867:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11689 00000100:00080000:3.0:1788574936.027873:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574936.027876:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574936.027884:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574936.027893:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569901184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574936.027900:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574936.027903:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574936.027904:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574936.027907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11689 00000100:00080000:3.0:1788574936.027909:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574936.027911:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574936.027914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574936.027921:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569901312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574936.027930:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574936.027932:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569901056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574936.027933:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574936.027936:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574936.027942:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11689 00000100:00080000:3.0:1788574936.027945:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574936.027947:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574936.027950:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574936.027959:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569901440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574936.027964:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574936.027966:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574936.027968:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574936.027970:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11689 00000100:00080000:3.0:1788574936.027972:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574936.027974:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574936.027977:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574936.027985:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569901568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574936.028016:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574936.028054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff920a529b1500 x1875444569901056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574936.028075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569901056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574936.028087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574936.028091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574936.028096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569901056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574936.028104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574936.028129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574936.028143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff920a529b1180 x1875444569901184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574936.028157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569901184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574936.028165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574936.028169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574936.028173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574936.028201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574936.028225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574936.028238:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff920a529b0e00 x1875444569901312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574936.028251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569901312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574936.028260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574936.028264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574936.028268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574936.028274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574936.028294:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574936.028306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff920a529b3b80 x1875444569901440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574936.028318:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569901440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574936.028328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574936.028332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574936.028335:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574936.028342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574936.028364:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574936.028375:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574936/real 1788574936] req@ffff920a529b0380 x1875444569901568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574936.028389:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569901568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574936.028398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574936.028411:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574936.028416:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574936.475748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574937.051709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574937.499792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574938.075708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574938.523760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574939.099683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574939.547750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574940.123760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574940.571773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574941.147670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574941.147698:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574941.147704:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574941.147709:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574941.147718:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11694 00000100:00080000:0.0:1788574941.147728:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574941.147736:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574941.147771:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569901696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574941.147796:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574941.147800:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574941.147802:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574941.147806:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11694 00000100:00080000:0.0:1788574941.147810:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574941.147815:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574941.147824:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574941.147836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569901696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574941.147838:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569901824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574941.147851:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574941.147855:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574941.147857:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574941.147861:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11694 00000100:00080000:0.0:1788574941.147864:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574941.147869:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574941.147873:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574941.147884:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569901952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574941.147893:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574941.147896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574941.147898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574941.147901:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11694 00000100:00080000:0.0:1788574941.147905:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574941.147908:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574941.147914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574941.147928:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569902080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574941.147939:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788574941.147940:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901824:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788574941.147948:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574941.147951:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574941.147956:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11694 00000100:00080000:0.0:1788574941.147959:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574941.147964:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574941.147970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574941.147980:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574941.147985:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569902208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788574941.148009:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574941.148030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574941.595757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574942.171768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574942.619761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574943.195772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574943.643779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574944.219866:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788574944.220027:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788574944.220066:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788574944.220096:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574944.220135:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574944.220146:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574944.220155:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788574944.220163:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788574944.220255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574944] req@ffff920a48350380 x1875444569902080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574944.220279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569902080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574944.220292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574944.220298:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574944.220302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574944.220311:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574944] req@ffff920a48353800 x1875444569902208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574944.220322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569902208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574944.220329:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574944.220332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574944.220335:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902208:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788574944.220341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574944] req@ffff920a48350a80 x1875444569901824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574944.220351:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569901824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574944.220364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574944.220369:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574944.220372:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574944.220378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574944] req@ffff920a48350700 x1875444569901952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574944.220388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569901952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574944.220395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574944.220398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574944.220401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569901952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574944.220407:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574941/real 1788574944] req@ffff920a48353480 x1875444569901696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574944.220418:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569901696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574944.220436:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574944.220440:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788574944.220443:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569901696:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788574944.667728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574945.243735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574945.691791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574946.267652:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574946.267683:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574946.267689:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574946.267692:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574946.267709:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11699 00000100:00080000:3.0:1788574946.267720:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574946.267729:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574946.267761:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569902336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574946.267780:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574946.267783:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574946.267784:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574946.267787:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11699 00000100:00080000:3.0:1788574946.267789:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574946.267792:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574946.267799:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574946.267808:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569902464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574946.267815:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788574946.267817:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569902336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574946.267818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574946.267820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574946.267827:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11699 00000100:00080000:3.0:1788574946.267830:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574946.267834:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574946.267838:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574946.267850:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569902592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574946.267858:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574946.267862:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574946.267864:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574946.267867:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11699 00000100:00080000:3.0:1788574946.267870:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574946.267874:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574946.267879:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574946.267889:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569902720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788574946.267898:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574946.267902:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574946.267906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574946.267908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574946.267913:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11699 00000100:00080000:3.0:1788574946.267919:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574946.267923:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574946.267929:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574946.267937:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff920a529b2300 x1875444569902336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574946.267941:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569902848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788574946.267972:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569902336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574946.267985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574946.267989:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574946.267993:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569902336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574946.268017:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574946.268041:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574946.268054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff920a529b0700 x1875444569902464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574946.268066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569902464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574946.268073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574946.268077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574946.268080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574946.268086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788574946.268105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574946.268116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff920a529b0000 x1875444569902592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574946.268128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569902592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574946.268135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574946.268138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574946.268142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788574946.268147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574946.268167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574946.268178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff920a529b1f80 x1875444569902720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574946.268203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569902720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574946.268212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574946.268216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574946.268219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788574946.268225:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788574946.268244:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788574946.268255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574946/real 1788574946] req@ffff920a529b3100 x1875444569902848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574946.268267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569902848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574946.268275:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788574946.268278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788574946.268281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569902848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574946.715747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574947.291710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574947.739794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788574948.315665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574948.763781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574949.339749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574949.787776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574950.363694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574950.811732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574951.387716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574951.387767:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574951.387773:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574951.387777:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574951.387797:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11704 00000100:00080000:3.0:1788574951.387806:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574951.387815:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574951.387848:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569902976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574951.387872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574951.387876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574951.387879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574951.387884:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11704 00000100:00080000:3.0:1788574951.387887:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574951.387894:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569902976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574951.387894:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574951.387907:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574951.387924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569903104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574951.387936:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574951.387939:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574951.387941:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574951.387946:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11704 00000100:00080000:3.0:1788574951.387951:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574951.387957:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574951.387957:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574951.387963:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574951.387978:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569903232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574951.387996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff920a529b2d80 x1875444569902976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574951.388013:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569902976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574951.388022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574951.388030:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574951.388034:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569902976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574951.388053:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574951.388057:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574951.388059:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574951.388064:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11704 00000100:00100000:2.0:1788574951.388067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903104:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574951.388070:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574951.388078:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574951.388085:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574951.388098:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569903360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574951.388102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574951.388117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff920a529b3480 x1875444569903104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574951.388128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569903104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574951.388136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574951.388139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574951.388142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574951.388146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574951.388161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574951.388168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff920a529b2a00 x1875444569903232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574951.388175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569903232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574951.388180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574951.388182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574951.388184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903232:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788574951.388240:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574951.388243:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574951.388245:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574951.388250:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11704 00000100:00080000:3.0:1788574951.388254:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574951.388259:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574951.388265:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574951.388275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574951.388303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574951.388319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff920a529b1c00 x1875444569903360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574951.388329:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569903360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574951.388335:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569903488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574951.388337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574951.388343:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574951.388348:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574951.388363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574951.388378:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574951.388384:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574951/real 1788574951] req@ffff920a529b2a00 x1875444569903488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574951.388391:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569903488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574951.388400:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574951.388403:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574951.388406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574951.835757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574952.411740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574952.859791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574953.435708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574953.883838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574954.459703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574954.907767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788574955.483692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574955.931773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574956.507704:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574956.507716:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574956.507720:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574956.507731:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11709 00000100:00100000:1.0:1788574956.507735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574956.507741:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574956.507750:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574956.507788:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569903616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574956.507814:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574956.507818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574956.507820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574956.507824:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11709 00000100:00080000:3.0:1788574956.507828:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574956.507833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574956.507843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569903616:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788574956.507844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574956.507862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569903744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574956.507873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574956.507877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574956.507879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574956.507884:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11709 00000100:00080000:3.0:1788574956.507895:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788574956.507900:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574956.507901:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574956.507909:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574956.507921:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569903872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574956.507930:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff920a529b3480 x1875444569903616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.507934:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574956.507948:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569903616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.507952:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574956.507957:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574956.507961:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574956.507965:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11709 00000100:00080000:2.0:1788574956.507967:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574956.507973:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569903616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574956.507973:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574956.507981:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574956.507987:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574956.507992:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574956.508001:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569904000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574956.508023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574956.508029:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574956.508033:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788574956.508036:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff920a529b2d80 x1875444569903744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.508037:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574956.508047:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11709 00000100:00100000:2.0:1788574956.508050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569903744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574956.508054:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788574956.508058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574956.508063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574956.508063:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788574956.508068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903744:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788574956.508073:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574956.508078:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574956.508089:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569904128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574956.508102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574956.508116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff920a529b3100 x1875444569903872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574956.508126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569903872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574956.508131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574956.508133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574956.508135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569903872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574956.508146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574956.508159:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574956.508165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff920a529b1f80 x1875444569904000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574956.508172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569904000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574956.508176:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574956.508178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574956.508180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574956.508183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574956.508192:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574956.508197:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574956/real 1788574956] req@ffff920a529b0000 x1875444569904128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574956.508203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569904128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574956.508208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574956.508209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574956.508211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574956.955768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574957.531711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574957.979744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574958.555692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574959.003742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574959.579701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574960.027797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574960.603668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574961.051749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574961.627674:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574961.627682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574961.627682:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574961.627687:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574961.627698:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11715 00000100:00080000:3.0:1788574961.627707:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574961.627713:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574961.627745:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569904256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574961.627766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574961.627769:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574961.627770:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574961.627773:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11715 00000100:00080000:3.0:1788574961.627775:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574961.627777:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574961.627785:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574961.627794:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569904384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574961.627802:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574961.627805:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574961.627806:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574961.627812:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11715 00000100:00080000:3.0:1788574961.627817:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574961.627819:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569904256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574961.627825:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574961.627835:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574961.627849:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569904512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574961.627855:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574961.627858:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574961.627859:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574961.627861:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11715 00000100:00080000:3.0:1788574961.627863:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574961.627866:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574961.627871:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574961.627881:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569904640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574961.627887:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574961.627890:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574961.627892:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574961.627893:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11715 00000100:00080000:3.0:1788574961.627895:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574961.627897:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574961.627900:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574961.627906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574961.627910:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569904768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574961.627951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574961/real 1788574961] req@ffff920a529b0700 x1875444569904256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574961.627968:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569904256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574961.627986:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574961.627990:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574961.627993:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569904256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574961.628017:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574961.628035:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574961.628044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574961/real 1788574961] req@ffff920a529b2300 x1875444569904384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574961.628052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569904384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574961.628062:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574961.628065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574961.628067:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574961.628071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574961.628082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574961.628087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574961/real 1788574961] req@ffff920a529b0380 x1875444569904512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574961.628094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569904512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574961.628100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574961.628102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574961.628103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574961.628107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574961.628118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574961.628124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574961/real 1788574961] req@ffff920a529b3b80 x1875444569904640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574961.628131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569904640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574961.628136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574961.628138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574961.628141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574961.628145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574961.628157:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574961.628162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574961/real 1788574961] req@ffff920a529b0e00 x1875444569904768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574961.628169:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569904768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574961.628174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574961.628176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574961.628178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569904768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574962.075737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574962.651705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574963.099756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574963.675753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574964.123720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574964.699695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574965.147750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574965.723671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574966.171727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574966.747699:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574966.747701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574966.747711:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574966.747716:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574966.747725:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11720 00000100:00080000:3.0:1788574966.747735:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574966.747742:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574966.747778:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569904896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574966.747814:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574966.747818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574966.747820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574966.747823:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11720 00000100:00080000:3.0:1788574966.747828:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574966.747832:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574966.747842:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574966.747857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569904896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574966.747860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569905024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574966.747872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574966.747876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574966.747878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574966.747882:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11720 00000100:00080000:3.0:1788574966.747885:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574966.747890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574966.747895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574966.747908:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569905152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574966.747917:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574966.747918:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574966.747924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574966.747927:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574966.747932:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11720 00000100:00080000:3.0:1788574966.747937:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574966.747942:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574966.747947:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574966.747952:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574966/real 1788574966] req@ffff920a529b1180 x1875444569904896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574966.747962:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569905280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574966.747974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569904896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574966.747985:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788574966.747991:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574966.747991:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574966.747994:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788574966.747998:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574966.748004:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11720 00000100:00100000:2.0:1788574966.748005:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569904896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574966.748011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574966.748017:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574966.748022:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574966.748031:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788574966.748035:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569905408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788574966.748056:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574966.748070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574966/real 1788574966] req@ffff920a529b1500 x1875444569905024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574966.748080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569905024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574966.748088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574966.748090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574966.748093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574966.748098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574966.748111:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574966.748119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574966/real 1788574966] req@ffff920a529b0e00 x1875444569905152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574966.748128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569905152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574966.748135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574966.748138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574966.748140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574966.748144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574966.748156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574966.748163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574966/real 1788574966] req@ffff920a529b3b80 x1875444569905280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574966.748172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569905280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574966.748179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574966.748182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574966.748184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574966.748195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574966.748208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574966.748215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574966/real 1788574966] req@ffff920a529b0380 x1875444569905408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574966.748224:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569905408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574966.748231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574966.748233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574966.748236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574967.195761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574967.771658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574968.219771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574968.795686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574969.243781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574969.819672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574970.267772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574970.843700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574971.291766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574971.867703:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574971.867718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788574971.867721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574971.867722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574971.867734:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11725 00000100:00080000:3.0:1788574971.867744:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574971.867750:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574971.867780:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569905536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574971.867800:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574971.867802:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574971.867804:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574971.867806:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11725 00000100:00080000:3.0:1788574971.867808:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574971.867811:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574971.867819:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574971.867826:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569905664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574971.867832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574971.867835:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574971.867836:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574971.867838:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11725 00000100:00080000:3.0:1788574971.867840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574971.867842:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574971.867845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574971.867851:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569905536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574971.867853:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569905792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574971.867865:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574971.867867:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574971.867868:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574971.867870:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11725 00000100:00080000:3.0:1788574971.867872:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574971.867874:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574971.867877:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574971.867883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569905920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574971.867889:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574971.867891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574971.867892:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574971.867894:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11725 00000100:00080000:3.0:1788574971.867909:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574971.867911:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574971.867914:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574971.867920:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574971.867921:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569906048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574971.867956:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574971/real 1788574971] req@ffff920a529b2300 x1875444569905536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574971.867973:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569905536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574971.867982:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574971.867986:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574971.867989:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569905536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574971.867997:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574971.868020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574971.868032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574971/real 1788574971] req@ffff920a529b0700 x1875444569905664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574971.868043:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569905664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574971.868051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574971.868054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574971.868057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574971.868081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574971.868100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574971.868111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574971/real 1788574971] req@ffff920a529b0000 x1875444569905792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574971.868121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569905792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574971.868128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574971.868131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574971.868134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574971.868139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574971.868158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574971.868168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574971/real 1788574971] req@ffff920a529b1f80 x1875444569905920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574971.868179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569905920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574971.868186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574971.868188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574971.868191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569905920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574971.868197:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574971.868215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574971.868225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574971/real 1788574971] req@ffff920a529b3100 x1875444569906048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574971.868245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569906048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788574996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574971.868253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574971.868256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574971.868259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574972.315799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574972.891688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574973.339761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574973.915685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574974.363764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574974.939680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574975.387770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574975.963703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574976.411759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574976.987684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788574976.987738:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574976.987749:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788574976.987754:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574976.987764:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11730 00000100:00080000:0.0:1788574976.987775:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574976.987782:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788574976.987818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569906176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574976.987849:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574976.987853:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788574976.987855:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574976.987860:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11730 00000100:00080000:0.0:1788574976.987865:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574976.987871:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574976.987881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574976.987892:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569906176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788574976.987913:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569906304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574976.987926:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574976.987930:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788574976.987931:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574976.987936:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11730 00000100:00080000:0.0:1788574976.987940:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574976.987944:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574976.987949:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788574976.987961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569906432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574976.987969:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574976.987971:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788574976.987973:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788574976.987978:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11730 00000100:00080000:0.0:1788574976.987982:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574976.987985:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788574976.987993:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788574976.987998:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788574976.988017:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569906560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788574976.988029:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788574976.988032:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788574976.988035:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574976.988038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574976/real 1788574976] req@ffff920a48353b80 x1875444569906176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788574976.988048:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11730 00000100:00080000:0.0:1788574976.988056:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788574976.988064:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788574976.988070:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574976.988075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569906176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788574976.988089:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569906688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574976.988118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574976.988139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574976.988145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569906176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574976.988173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574976.988203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574976.988219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574976/real 1788574976] req@ffff920a48351500 x1875444569906304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574976.988235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569906304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574976.988245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574976.988250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574976.988254:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574976.988262:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574976.988285:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574976.988297:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574976/real 1788574976] req@ffff920a48351c00 x1875444569906432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574976.988310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569906432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574976.988320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574976.988324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574976.988327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574976.988335:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574976.988360:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574976.988374:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574976/real 1788574976] req@ffff920a48353480 x1875444569906560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574976.988386:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569906560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574976.988395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574976.988398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574976.988402:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574976.988408:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574976.988426:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574976.988434:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574976/real 1788574976] req@ffff920a48350700 x1875444569906688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574976.988441:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569906688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575001 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574976.988446:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574976.988448:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574976.988450:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574977.435742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574978.011717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574978.459762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574979.035736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574979.483789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574980.059718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574980.507706:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574981.083731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574981.531743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574982.107701:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788574982.107703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574982.107716:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574982.107721:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574982.107735:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11735 00000100:00080000:3.0:1788574982.107747:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574982.107757:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574982.107808:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569906816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574982.107838:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574982.107842:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574982.107844:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574982.107849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11735 00000100:00080000:3.0:1788574982.107853:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574982.107858:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574982.107870:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574982.107872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569906816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574982.107886:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569906944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574982.107906:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574982.107909:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574982.107911:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574982.107916:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11735 00000100:00080000:3.0:1788574982.107919:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574982.107923:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574982.107928:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574982.107941:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569907072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574982.107951:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574982.107954:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574982.107956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574982.107960:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11735 00000800:00000100:2.0:1788574982.107966:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574982.107966:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574982.107974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574982.107980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574982.107991:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569907200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574982.108000:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574982.108004:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574982.108006:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574982.108010:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11735 00000100:00080000:3.0:1788574982.108015:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788574982.108016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff920a529b2d80 x1875444569906816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574982.108024:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574982.108031:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574982.108039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569906816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574982.108045:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569907328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574982.108053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574982.108059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574982.108063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569906816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574982.108088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574982.108106:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574982.108115:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff920a529b3480 x1875444569906944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574982.108162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569906944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574982.108172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574982.108176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574982.108180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569906944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574982.108187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574982.108217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574982.108231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff920a529b2a00 x1875444569907072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574982.108243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569907072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574982.108263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574982.108265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574982.108268:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574982.108274:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574982.108297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574982.108310:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff920a529b1c00 x1875444569907200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574982.108323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569907200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574982.108332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574982.108335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574982.108338:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574982.108345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574982.108366:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574982.108377:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574982/real 1788574982] req@ffff920a529b3100 x1875444569907328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574982.108386:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569907328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574982.108391:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574982.108393:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574982.108395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574982.555785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574983.131722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574983.579735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574984.155747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574984.603792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574985.179713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574985.627849:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574986.203758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574986.651794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574987.227665:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574987.227674:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788574987.227679:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574987.227688:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11740 00000100:00080000:3.0:1788574987.227698:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574987.227706:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574987.227744:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569907456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788574987.227751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574987.227773:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574987.227777:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574987.227779:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574987.227783:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11740 00000100:00080000:3.0:1788574987.227786:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574987.227791:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574987.227806:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574987.227818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569907584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574987.227828:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788574987.227831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569907456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574987.227832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574987.227835:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574987.227842:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11740 00000100:00080000:3.0:1788574987.227845:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574987.227847:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574987.227850:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574987.227860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569907712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574987.227867:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574987.227869:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574987.227870:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574987.227873:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11740 00000100:00080000:3.0:1788574987.227875:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574987.227877:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574987.227881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574987.227888:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569907840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574987.227894:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574987.227896:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574987.227897:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574987.227902:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11740 00000800:00000100:2.0:1788574987.227903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788574987.227905:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574987.227916:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574987.227923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574987.227931:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569907968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574987.227955:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff920a529b1f80 x1875444569907456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.227971:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569907456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574987.227980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574987.227983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574987.227986:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569907456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574987.228008:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574987.228029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574987.228041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff920a529b0000 x1875444569907584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.228050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569907584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574987.228057:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574987.228059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574987.228062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574987.228066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574987.228092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574987.228101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff920a529b0700 x1875444569907712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.228115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569907712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574987.228128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574987.228132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574987.228136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574987.228142:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574987.228172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574987.228187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff920a529b2300 x1875444569907840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.228199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569907840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574987.228206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574987.228209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574987.228212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574987.228221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574987.228233:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574987.228240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574987/real 1788574987] req@ffff920a529b0380 x1875444569907968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574987.228248:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569907968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574987.228254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574987.228256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574987.228258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569907968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574987.675844:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574988.251739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574988.699778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574989.275745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574989.723793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574990.299703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574990.747757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574991.323725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574991.771718:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574992.347702:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574992.347713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788574992.347715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574992.347718:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574992.347730:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11745 00000100:00080000:3.0:1788574992.347740:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574992.347747:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574992.347784:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569908096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574992.347804:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574992.347808:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574992.347810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574992.347814:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11745 00000100:00080000:3.0:1788574992.347818:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574992.347823:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574992.347834:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788574992.347847:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569908096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788574992.347850:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569908224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574992.347880:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574992.347884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788574992.347886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574992.347892:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11745 00000100:00080000:3.0:1788574992.347906:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574992.347910:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788574992.347916:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788574992.347920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574992.347935:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569908352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574992.347943:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574992.347946:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574992.347948:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574992.347951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574992] req@ffff920a529b3b80 x1875444569908096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574992.347955:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11745 00000100:00080000:3.0:1788574992.347960:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574992.347966:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574992.347971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788574992.347977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569908096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788574992.347981:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569908480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788574992.348003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788574992.348003:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574992.348010:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788574992.348013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788574992.348013:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788574992.348021:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569908096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574992.348021:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11745 00000100:00080000:3.0:1788574992.348027:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574992.348031:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574992.348033:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574992.348041:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569908608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574992.348048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574992.348075:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574992.348085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574992] req@ffff920a529b0e00 x1875444569908224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574992.348094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569908224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574992.348099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574992.348101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574992.348104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574992.348108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574992.348118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574992.348124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574992] req@ffff920a529b1500 x1875444569908352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574992.348132:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569908352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574992.348137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574992.348138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574992.348140:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574992.348143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574992.348154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574992.348159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574992] req@ffff920a529b1180 x1875444569908480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574992.348166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569908480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574992.348171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574992.348172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574992.348174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574992.348184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574992.348194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574992.348199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574992/real 1788574992] req@ffff920a529b0380 x1875444569908608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574992.348207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569908608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574992.348211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574992.348213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574992.348215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574992.795801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574993.371733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574993.819802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574994.395718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574994.843826:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574995.419692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574995.867761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574996.443680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574996.891771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788574997.467675:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574997.467684:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788574997.467685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788574997.467688:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574997.467698:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11750 00000100:00080000:3.0:1788574997.467707:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574997.467714:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788574997.467747:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569908736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574997.467770:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574997.467774:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788574997.467775:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574997.467778:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11750 00000100:00080000:3.0:1788574997.467781:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574997.467796:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574997.467806:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574997.467817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569908864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574997.467832:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574997.467837:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788574997.467838:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569908736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788574997.467839:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574997.467847:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11750 00000100:00080000:3.0:1788574997.467851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574997.467853:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574997.467858:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788574997.467867:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569908992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574997.467875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574997.467878:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788574997.467879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574997.467883:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11750 00000100:00080000:3.0:1788574997.467891:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574997.467894:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574997.467898:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788574997.467899:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788574997.467909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569909120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788574997.467922:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788574997.467925:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788574997.467927:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788574997.467930:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11750 00000100:00100000:2.0:1788574997.467933:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff920a529b2300 x1875444569908736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788574997.467934:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788574997.467941:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788574997.467948:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788574997.467959:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569909248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788574997.467961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569908736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574997.467985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574997.467989:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574997.467992:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569908736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788574997.468000:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574997.468016:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574997.468024:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff920a529b0700 x1875444569908864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574997.468032:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569908864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574997.468042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574997.468045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574997.468047:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574997.468065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788574997.468076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574997.468083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff920a529b0000 x1875444569908992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574997.468090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569908992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574997.468095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574997.468097:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574997.468099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569908992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788574997.468103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574997.468113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574997.468119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff920a529b1f80 x1875444569909120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574997.468126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569909120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574997.468131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574997.468133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574997.468135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788574997.468139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788574997.468149:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788574997.468154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788574997/real 1788574997] req@ffff920a529b3100 x1875444569909248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788574997.468161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569909248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788574997.468166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788574997.468168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788574997.468170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788574997.915747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574998.491691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574998.939796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788574999.515692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788574999.963785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575000.539728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575000.987758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575001.563734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575002.011759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575002.587702:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575002.587709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575002.587714:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575002.587719:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.587731:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11755 00000100:00080000:1.0:1788575002.587741:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.587750:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575002.587786:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569909376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575002.587808:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.587811:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575002.587813:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.587816:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11755 00000100:00080000:1.0:1788575002.587819:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.587823:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575002.587832:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575002.587841:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569909376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575002.587855:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569909504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575002.587869:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.587874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575002.587876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.587881:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11755 00000100:00080000:1.0:1788575002.587885:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.587890:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788575002.587896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788575002.587897:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575002.587916:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569909632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575002.587926:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff920a52891f80 x1875444569909376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575002.587929:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.587940:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1788575002.587950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569909376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575002.587956:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575002.587966:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11755 00000100:00080000:1.0:1788575002.587969:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.587976:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575002.587982:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575002.587982:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575002.587992:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575002.588000:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569909760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575002.588000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569909376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575002.588030:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575002.588034:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575002.588037:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575002.588042:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909504:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788575002.588046:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11755 00000100:00080000:1.0:1788575002.588053:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575002.588058:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575002.588065:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575002.588069:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575002.588088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff920a52891500 x1875444569909504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575002.588089:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569909888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575002.588112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569909504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575002.588123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575002.588126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575002.588129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575002.588135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575002.588147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575002.588154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff920a52892680 x1875444569909632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575002.588162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569909632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575002.588169:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575002.588171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575002.588175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575002.588181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575002.588202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575002.588213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff920a52892a00 x1875444569909760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575002.588226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569909760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575002.588236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575002.588239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575002.588243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575002.588258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575002.588273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575002.588281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575002/real 1788575002] req@ffff920a52890a80 x1875444569909888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575002.588289:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569909888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575002.588294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575002.588297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575002.588299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569909888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575003.035720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575003.611725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575004.059769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575004.635656:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575005.083738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575005.659702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575006.107789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575006.683692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575007.131800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575007.707725:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575007.707727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575007.707739:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575007.707745:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575007.707758:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11761 00000100:00080000:1.0:1788575007.707771:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575007.707781:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575007.707844:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569910016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575007.707880:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575007.707884:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575007.707886:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575007.707891:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11761 00000100:00080000:1.0:1788575007.707895:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575007.707900:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575007.707912:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575007.707928:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569910144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575007.707939:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575007.707942:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575007.707944:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575007.707949:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11761 00000100:00080000:1.0:1788575007.707952:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575007.707956:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575007.707971:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575007.707985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569910016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575007.707991:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569910272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575007.708010:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575007.708012:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575007.708014:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575007.708017:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11761 00000100:00080000:1.0:1788575007.708019:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575007.708023:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575007.708028:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575007.708039:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569910400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575007.708045:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575007.708048:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575007.708050:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575007.708061:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11761 00000100:00080000:1.0:1788575007.708064:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575007.708067:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575007.708070:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575007.708080:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569910528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575007.708116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575007.708155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575007.708189:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575007.708231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575008.155822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575008.731732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575009.179769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575009.755709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575010.203820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575010.779674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788575010.779936:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788575010.779983:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575010.780000:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575010.780048:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575010.780060:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575010.780074:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575010.780087:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575010.780100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575007/real 1788575010] req@ffff920a52890a80 x1875444569910272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575010.780138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569910272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575010.780158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575010.780165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575010.780170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575010.780179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575007/real 1788575010] req@ffff920a52892a00 x1875444569910400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575010.780193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569910400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575010.780202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575010.780206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575010.780210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575010.780218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575007/real 1788575010] req@ffff920a52892680 x1875444569910528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575010.780231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569910528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575010.780244:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575010.780248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575010.780252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910528:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575010.780260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575007/real 1788575010] req@ffff920a52890000 x1875444569910016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575010.780275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569910016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575010.780284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575010.780288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575010.780331:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569910016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575010.780340:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575007/real 1788575010] req@ffff920a52893b80 x1875444569910144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575010.780354:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569910144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575010.780363:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575010.780367:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575010.780370:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910144:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788575011.227776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575011.803670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575012.251797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575012.827739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575012.827821:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575012.827829:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575012.827834:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575012.827847:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11766 00000100:00080000:0.0:1788575012.827860:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575012.827869:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575012.827914:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569910656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575012.827942:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575012.827946:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575012.827948:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575012.827953:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11766 00000100:00080000:0.0:1788575012.827967:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575012.827974:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575012.827987:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575012.827995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569910656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575012.828009:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569910784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575012.828024:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575012.828028:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575012.828030:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575012.828036:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11766 00000100:00080000:0.0:1788575012.828040:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575012.828045:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575012.828051:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575012.828065:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569910912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575012.828086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575012.828101:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575012.828107:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575012.828109:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575012.828115:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11766 00000100:00080000:0.0:1788575012.828119:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575012.828125:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575012.828131:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575012.828133:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575012/real 1788575012] req@ffff920a48350a80 x1875444569910656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575012.828148:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569911040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575012.828161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569910656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575012.828162:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575012.828172:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575012.828175:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575012.828179:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11766 00000100:00080000:0.0:1788575012.828183:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575012.828187:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575012.828192:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575012.828201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575012.828207:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569911168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575012.828209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575012.828220:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569910656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575012.828265:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575012.828293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575012.828309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575012/real 1788575012] req@ffff920a48353800 x1875444569910784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575012.828325:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569910784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575012.828335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575012.828353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575012.828357:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575012.828364:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575012.828388:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575012.828401:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575012/real 1788575012] req@ffff920a48350380 x1875444569910912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575012.828413:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569910912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575012.828423:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575012.828427:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575012.828431:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569910912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575012.828437:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575012.828462:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575012.828475:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575012/real 1788575012] req@ffff920a48350700 x1875444569911040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575012.828488:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569911040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575012.828497:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575012.828500:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575012.828504:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575012.828510:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575012.828539:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575012.828551:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575012/real 1788575012] req@ffff920a48353480 x1875444569911168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575012.828563:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569911168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575012.828570:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575012.828573:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575012.828577:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575013.275762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575013.851712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575014.299776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575014.875665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575015.323822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575015.899708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575016.347795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575016.923672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575017.371747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575017.947720:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575017.947731:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575017.947736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575017.947746:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11771 00000100:00080000:3.0:1788575017.947758:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575017.947767:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575017.947769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788575017.947811:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569911296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575017.947835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575017.947838:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575017.947839:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575017.947842:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11771 00000100:00080000:3.0:1788575017.947844:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575017.947847:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575017.947858:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575017.947870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569911424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575017.947878:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575017.947880:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575017.947881:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575017.947885:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11771 00000100:00080000:3.0:1788575017.947886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575017.947889:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575017.947892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575017.947895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569911296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575017.947901:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569911552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575017.947909:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575017.947912:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575017.947913:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575017.947916:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11771 00000100:00080000:3.0:1788575017.947918:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575017.947927:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575017.947931:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575017.947942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569911680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575017.947948:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575017.947951:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575017.947953:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575017.947955:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11771 00000100:00080000:3.0:1788575017.947960:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575017.947963:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575017.947966:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575017.947973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575017.947976:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569911808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575017.948014:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575017/real 1788575017] req@ffff920a529b1c00 x1875444569911296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575017.948030:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569911296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575017.948044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575017.948048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575017.948051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569911296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575017.948057:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575017.948074:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575017.948086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575017/real 1788575017] req@ffff920a529b2a00 x1875444569911424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575017.948098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569911424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575017.948106:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575017.948109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575017.948112:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575017.948143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575017.948165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575017.948188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575017/real 1788575017] req@ffff920a529b3480 x1875444569911552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575017.948203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569911552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575017.948212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575017.948217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575017.948221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575017.948228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575017.948250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575017.948262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575017/real 1788575017] req@ffff920a529b2d80 x1875444569911680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575017.948276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569911680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575017.948285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575017.948288:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575017.948292:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575017.948299:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575017.948322:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575017.948333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575017/real 1788575017] req@ffff920a529b3100 x1875444569911808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575017.948345:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569911808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575042 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575017.948353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575017.948356:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575017.948358:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569911808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575018.395786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575018.971694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575019.419809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575019.995716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575020.443750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575021.019747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575021.467814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575022.043706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575022.491791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575023.067720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575023.067720:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.067734:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575023.067740:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.067754:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11776 00000100:00080000:3.0:1788575023.067767:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.067776:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575023.067817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569911936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575023.067844:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.067847:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575023.067849:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.067852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11776 00000100:00080000:3.0:1788575023.067855:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.067858:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575023.067868:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575023.067886:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569912064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575023.067898:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569911936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575023.067913:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.067918:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575023.067921:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.067928:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11776 00000100:00080000:3.0:1788575023.067933:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.067938:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575023.067944:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575023.067961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569912192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575023.067974:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.067979:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575023.067981:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.067984:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11776 00000100:00080000:3.0:1788575023.067989:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575023.067992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575023.067996:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575023.068003:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575023.068013:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569912320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575023.068043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff920a529b1f80 x1875444569911936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575023.068059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569911936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575023.068068:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575023.068072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575023.068076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569911936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575023.068077:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575023.068083:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575023.068085:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575023.068089:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11776 00000100:00080000:3.0:1788575023.068093:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575023.068097:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575023.068100:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575023.068106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575023.068113:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569912448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575023.068131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575023.068143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff920a529b0000 x1875444569912064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575023.068152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569912064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575023.068158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575023.068160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575023.068162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575023.068165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575023.068177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575023.068183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff920a529b0700 x1875444569912192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575023.068190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569912192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575023.068199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575023.068202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575023.068204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575023.068207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575023.068219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575023.068227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff920a529b2300 x1875444569912320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575023.068233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569912320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575023.068239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575023.068241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575023.068243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575023.068254:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575023.068268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575023.068276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575023/real 1788575023] req@ffff920a529b0380 x1875444569912448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575023.068284:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569912448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575023.068290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575023.068292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575023.068295:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575023.515795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575024.091692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575024.539833:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575025.115828:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575025.563806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575026.139714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575026.587826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575027.163725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575027.611842:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575028.187748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575028.187766:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.187772:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575028.187777:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.187787:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11781 00000100:00080000:3.0:1788575028.187798:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.187805:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575028.187849:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569912576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.187875:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.187879:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575028.187880:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.187884:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11781 00000100:00080000:3.0:1788575028.187886:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.187888:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575028.187899:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575028.187908:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569912704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.187917:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575028.187918:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569912576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575028.187920:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575028.187924:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.187932:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11781 00000100:00080000:3.0:1788575028.187934:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.187937:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575028.187940:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575028.187949:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569912832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.187955:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.187957:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575028.187958:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.187961:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11781 00000100:00080000:3.0:1788575028.187963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.187966:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575028.187969:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575028.187975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569912960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575028.187981:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575028.187983:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575028.187985:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575028.187987:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11781 00000100:00080000:3.0:1788575028.187992:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575028.187995:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575028.187999:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575028.188005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575028.188010:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569913088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575028.188043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff920a529b1180 x1875444569912576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575028.188059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569912576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575028.188068:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575028.188071:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575028.188074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569912576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575028.188100:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575028.188123:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575028.188137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff920a529b1500 x1875444569912704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575028.188151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569912704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575028.188158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575028.188161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575028.188163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575028.188168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575028.188183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575028.188194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff920a529b0e00 x1875444569912832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575028.188206:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569912832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575028.188219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575028.188221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575028.188224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575028.188230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575028.188263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575028.188275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff920a529b3b80 x1875444569912960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575028.188286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569912960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575028.188293:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575028.188296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575028.188299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569912960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575028.188305:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575028.188323:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575028.188333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575028/real 1788575028] req@ffff920a529b0380 x1875444569913088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575028.188344:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569913088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575028.188352:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575028.188355:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575028.188359:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575028.635763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575029.211779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575029.659814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575030.235713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575030.683785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575031.259703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575031.707822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575032.283696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575032.731820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575033.307741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575033.307761:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.307769:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575033.307775:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.307788:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11786 00000100:00080000:3.0:1788575033.307801:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.307821:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575033.307874:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569913216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.307904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.307907:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.307909:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.307912:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11786 00000100:00080000:3.0:1788575033.307915:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.307918:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575033.307927:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.307940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569913344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.307947:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.307950:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575033.307952:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.307955:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11786 00000100:00080000:3.0:1788575033.307958:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.307963:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575033.307964:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569913216:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575033.307967:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575033.307978:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569913472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.307985:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.307987:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575033.307988:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.307990:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11786 00000100:00080000:3.0:1788575033.307992:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.307995:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575033.307998:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575033.308005:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569913600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575033.308011:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575033.308014:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575033.308015:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575033.308017:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11786 00000100:00080000:3.0:1788575033.308041:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575033.308043:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575033.308046:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575033.308055:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569913728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575033.308058:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575033.308106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff920a529b2300 x1875444569913216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.308129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569913216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.308141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.308146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575033.308151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569913216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575033.308176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575033.308208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575033.308222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff920a529b0700 x1875444569913344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.308236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569913344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.308245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.308249:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575033.308252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575033.308281:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575033.308303:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575033.308315:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff920a529b0000 x1875444569913472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.308328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569913472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.308336:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.308340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575033.308344:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575033.308350:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575033.308372:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575033.308383:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff920a529b1f80 x1875444569913600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.308396:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569913600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.308404:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.308408:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575033.308411:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575033.308417:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575033.308449:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575033.308462:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575033/real 1788575033] req@ffff920a529b3100 x1875444569913728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575033.308475:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569913728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575033.308485:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575033.308498:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575033.308502:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575033.755859:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575034.331715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575034.779732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575035.355704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575035.803808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575036.379680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575036.827819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575037.403764:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575037.851736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575038.427724:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575038.427732:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575038.427736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575038.427748:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11791 00000100:00100000:1.0:1788575038.427750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575038.427762:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575038.427771:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575038.427802:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569913856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575038.427826:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575038.427828:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575038.427830:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575038.427833:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11791 00000100:00080000:0.0:1788575038.427835:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575038.427837:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575038.427844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575038.427857:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569913984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575038.427864:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575038.427867:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575038.427868:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575038.427871:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11791 00000100:00080000:0.0:1788575038.427873:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575038.427878:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575038.427884:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575038.427890:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569913856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575038.427898:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569914112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575038.427914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575038.427918:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575038.427920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575038.427923:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11791 00000100:00080000:0.0:1788575038.427936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575038.427941:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575038.427946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575038.427958:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569914240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575038.427966:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575038.427971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575038.427972:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575038.427975:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575038.427980:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11791 00000100:00080000:0.0:1788575038.427983:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575038.427987:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575038.427993:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575038.428005:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569914368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575038.428013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff920a48351c00 x1875444569913856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.428034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569913856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.428045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.428048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575038.428050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569913856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575038.428056:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575038.428073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575038.428088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff920a48351500 x1875444569913984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.428096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569913984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.428101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.428103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575038.428105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569913984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575038.428124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575038.428138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575038.428147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff920a48353b80 x1875444569914112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.428158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569914112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.428165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.428168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575038.428171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575038.428176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575038.428196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575038.428207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff920a48353480 x1875444569914240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.428218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569914240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.428224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.428227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575038.428230:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575038.428236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575038.428253:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575038.428260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575038/real 1788575038] req@ffff920a48350700 x1875444569914368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575038.428267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569914368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575038.428272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575038.428273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575038.428275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575038.875786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575039.451717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575039.899772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575040.475685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575040.923732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575041.499659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575041.947707:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575042.523715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575042.971759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575043.547712:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575043.547713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575043.547724:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575043.547736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575043.547749:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11796 00000100:00080000:3.0:1788575043.547763:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575043.547773:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575043.547825:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569914496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575043.547849:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575043.547852:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575043.547853:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575043.547856:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11796 00000100:00080000:3.0:1788575043.547859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575043.547862:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575043.547872:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575043.547883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569914624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575043.547887:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569914496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575043.547893:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575043.547897:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575043.547898:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575043.547901:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11796 00000100:00080000:3.0:1788575043.547903:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575043.547905:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575043.547909:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575043.547917:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569914752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575043.547923:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575043.547926:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575043.547927:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575043.547929:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11796 00000100:00080000:3.0:1788575043.547931:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575043.547933:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575043.547936:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575043.547941:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569914880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575043.547946:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575043.547948:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575043.547949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575043.547951:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11796 00000100:00080000:3.0:1788575043.547974:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575043.547976:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575043.547979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575043.547990:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569915008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575043.547991:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575043.548035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff920a529b2d80 x1875444569914496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.548062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569914496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.548070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575043.548073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575043.548076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569914496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575043.548107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575043.548131:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575043.548143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff920a529b3480 x1875444569914624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.548155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569914624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.548171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575043.548174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575043.548178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575043.548184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575043.548202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575043.548211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff920a529b2a00 x1875444569914752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.548223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569914752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.548231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575043.548235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575043.548238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575043.548243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575043.548262:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575043.548271:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff920a529b1c00 x1875444569914880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.548283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569914880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.548291:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575043.548294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575043.548298:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569914880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575043.548304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575043.548321:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575043.548331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575043/real 1788575043] req@ffff920a529b3100 x1875444569915008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575043.548342:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569915008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575043.548350:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575043.548353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575043.548356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575043.995796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575044.571743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575045.019764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575045.595685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575046.043775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575046.619692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575047.067754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575047.643705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575048.091791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575048.667724:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575048.667728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575048.667737:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575048.667743:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575048.667756:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11802 00000100:00080000:0.0:1788575048.667774:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575048.667782:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575048.667820:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569915136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575048.667841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575048.667844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575048.667845:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575048.667852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11802 00000100:00080000:0.0:1788575048.667855:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575048.667857:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575048.667866:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575048.667875:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569915264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575048.667882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575048.667884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575048.667885:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575048.667888:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11802 00000100:00080000:0.0:1788575048.667890:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575048.667892:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575048.667894:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575048.667899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569915392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575048.667907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575048.667909:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575048.667910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575048.667912:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11802 00000100:00080000:0.0:1788575048.667914:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575048.667921:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575048.667925:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569915136:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575048.667926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575048.667942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569915520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575048.667953:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575048.667957:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575048.667959:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575048.667964:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11802 00000100:00080000:0.0:1788575048.667969:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575048.667974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575048.667979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575048.667990:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569915648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575048.667998:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575048.668042:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575048/real 1788575048] req@ffff920a48350380 x1875444569915136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575048.668060:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569915136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575048.668070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575048.668074:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575048.668078:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569915136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575048.668085:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575048.668108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575048.668120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575048/real 1788575048] req@ffff920a48353800 x1875444569915264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575048.668131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569915264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575048.668138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575048.668141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575048.668144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575048.668191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575048.668212:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575048.668224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575048/real 1788575048] req@ffff920a48350a80 x1875444569915392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575048.668234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569915392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575048.668241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575048.668244:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575048.668247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575048.668253:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575048.668271:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575048.668281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575048/real 1788575048] req@ffff920a48350700 x1875444569915520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575048.668291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569915520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575048.668299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575048.668302:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575048.668305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575048.668311:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575048.668329:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575048.668353:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575048/real 1788575048] req@ffff920a48353480 x1875444569915648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575048.668365:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569915648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575048.668372:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575048.668375:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575048.668379:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575049.115814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575049.691671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575050.139728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575050.715716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575051.163731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575051.739842:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575052.187745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575052.763718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575053.211808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575053.787693:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575053.787701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575053.787704:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575053.787709:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575053.787723:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11807 00000100:00080000:3.0:1788575053.787734:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575053.787742:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575053.787780:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569915776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575053.787808:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575053.787811:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575053.787813:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575053.787817:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11807 00000100:00080000:3.0:1788575053.787820:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575053.787825:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575053.787835:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575053.787849:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569915904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575053.787860:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575053.787864:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575053.787866:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575053.787872:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11807 00000100:00100000:2.0:1788575053.787875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569915776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575053.787876:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575053.787882:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575053.787888:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575053.787897:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569916032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575053.787904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575053.787906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575053.787908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575053.787911:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11807 00000100:00080000:3.0:1788575053.787913:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575053.787915:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575053.787919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575053.787925:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569916160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575053.787931:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575053.787933:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575053.787935:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575053.787937:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11807 00000100:00080000:3.0:1788575053.787939:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575053.787940:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575053.787943:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575053.787948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575053.787952:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569916288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575053.787988:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575053/real 1788575053] req@ffff920a529b1f80 x1875444569915776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575053.788008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569915776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575053.788018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575053.788032:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575053.788036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569915776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575053.788045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575053.788068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575053.788081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575053/real 1788575053] req@ffff920a529b0000 x1875444569915904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575053.788093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569915904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575053.788102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575053.788105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575053.788108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569915904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575053.788132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575053.788152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575053.788163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575053/real 1788575053] req@ffff920a529b0700 x1875444569916032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575053.788189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569916032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575053.788198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575053.788201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575053.788204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575053.788210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575053.788231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575053.788242:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575053/real 1788575053] req@ffff920a529b2300 x1875444569916160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575053.788254:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569916160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575053.788263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575053.788266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575053.788269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575053.788275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575053.788325:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575053.788337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575053/real 1788575053] req@ffff920a529b0380 x1875444569916288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575053.788350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569916288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575053.788358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575053.788361:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575053.788364:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575054.235735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575054.811713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575055.259735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575055.835670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575056.283692:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575056.859683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575057.307733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575057.883691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575058.331742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575058.907687:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575058.907699:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575058.907703:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575058.907710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575058.907713:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11812 00000100:00080000:3.0:1788575058.907724:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575058.907732:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575058.907765:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569916416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575058.907789:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575058.907793:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575058.907794:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575058.907799:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11812 00000100:00080000:3.0:1788575058.907802:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575058.907807:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575058.907817:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575058.907828:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569916416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575058.907832:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569916544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575058.907844:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575058.907847:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575058.907850:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575058.907855:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11812 00000100:00080000:3.0:1788575058.907859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575058.907863:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575058.907869:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575058.907881:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569916672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575058.907891:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575058.907896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575058.907897:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575058.907901:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575058.907906:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11812 00000100:00080000:3.0:1788575058.907910:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575058.907914:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575058.907919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575058.907935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575058/real 1788575058] req@ffff920a529b3b80 x1875444569916416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575058.907961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569916416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575058.907971:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575058.907976:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575058.907981:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569916416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575058.907986:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569916800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575058.907994:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575058.907996:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575058.907998:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575058.908000:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11812 00000100:00100000:0.0:1788575058.908006:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575058.908011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575058.908015:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575058.908019:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575058.908029:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569916928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575058.908037:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575058.908055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575058/real 1788575058] req@ffff920a529b0e00 x1875444569916544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575058.908067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569916544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575058.908076:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575058.908079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575058.908082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575058.908088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575058.908107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575058.908118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575058/real 1788575058] req@ffff920a529b1500 x1875444569916672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575058.908129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569916672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575058.908143:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575058.908147:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575058.908151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575058.908158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575058.908178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575058.908189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575058/real 1788575058] req@ffff920a529b1180 x1875444569916800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575058.908203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569916800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575058.908211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575058.908215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575058.908218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575058.908241:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575058.908263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575058.908276:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575058/real 1788575058] req@ffff920a529b3b80 x1875444569916928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575058.908292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569916928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575058.908302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575058.908306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575058.908311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569916928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575059.355749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575059.931719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575060.379785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575060.955696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575061.403737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575061.979688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575062.427815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575063.003697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575063.451822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575064.027722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575064.027841:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575064.027848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575064.027852:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575064.027862:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11817 00000100:00080000:3.0:1788575064.027872:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575064.027880:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575064.027922:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569917056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575064.027948:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575064.027951:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575064.027952:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575064.027955:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11817 00000100:00080000:3.0:1788575064.027958:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575064.027960:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575064.027970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575064.027979:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569917184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575064.027985:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575064.027988:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575064.027990:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575064.027993:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11817 00000100:00080000:3.0:1788575064.027995:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575064.027998:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575064.028001:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575064.028007:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569917312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575064.028014:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575064.028018:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575064.028021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569917056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575064.028021:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575064.028026:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11817 00000100:00080000:3.0:1788575064.028031:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575064.028034:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575064.028038:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575064.028045:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569917440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575064.028051:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575064.028054:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575064.028055:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575064.028057:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11817 00000100:00080000:3.0:1788575064.028059:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575064.028061:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575064.028064:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575064.028072:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569917568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575064.028114:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575064.028166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff920a529b0380 x1875444569917056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.028186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569917056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.028206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.028211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575064.028215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569917056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575064.028223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575064.028247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575064.028268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff920a529b2300 x1875444569917184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.028281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569917184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.028290:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.028294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575064.028297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575064.028327:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575064.028348:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575064.028360:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff920a529b0700 x1875444569917312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.028373:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569917312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.028382:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.028386:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575064.028390:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575064.028396:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575064.028417:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575064.028429:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff920a529b0000 x1875444569917440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.028441:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569917440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.028450:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.028454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575064.028458:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575064.028464:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575064.028486:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575064.028498:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575064/real 1788575064] req@ffff920a529b1f80 x1875444569917568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575064.028509:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569917568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575064.028551:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575064.028555:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575064.028559:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575064.475769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575065.051740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575065.499751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575066.075725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575066.523801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575067.099735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575067.547808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575068.123760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575068.571793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575069.147746:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575069.147752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575069.147765:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575069.147771:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575069.147785:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11822 00000100:00080000:0.0:1788575069.147800:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575069.147810:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575069.147862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569917696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575069.147897:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575069.147902:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575069.147904:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575069.147909:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11822 00000100:00080000:0.0:1788575069.147915:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575069.147920:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575069.147934:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575069.147935:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569917696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575069.147950:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569917824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575069.147963:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575069.147967:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575069.147971:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575069.147975:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11822 00000100:00080000:0.0:1788575069.147980:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575069.147984:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575069.147991:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575069.148004:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569917952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575069.148014:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575069.148020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575069.148022:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575069.148026:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575069.148032:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11822 00000100:00080000:0.0:1788575069.148037:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575069.148042:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575069.148049:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575069.148063:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569918080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575069.148071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff920a48353b80 x1875444569917696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.148076:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575069.148085:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575069.148089:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575069.148095:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11822 00000100:00080000:0.0:1788575069.148099:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575069.148101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569917696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575069.148106:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575069.148116:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575069.148119:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575069.148127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575069.148133:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569918208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575069.148135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569917696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575069.148175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575069.148203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575069.148218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff920a48351500 x1875444569917824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575069.148228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569917824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575069.148234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575069.148236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575069.148239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575069.148243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575069.148258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575069.148265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff920a48351c00 x1875444569917952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575069.148272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569917952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575069.148283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575069.148285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575069.148287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569917952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575069.148291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575069.148306:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575069.148315:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff920a48353480 x1875444569918080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575069.148327:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569918080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575069.148337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575069.148341:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575069.148344:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575069.148351:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575069.148370:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575069.148379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575069/real 1788575069] req@ffff920a48350700 x1875444569918208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575069.148387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569918208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575069.148392:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575069.148394:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575069.148397:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575069.595818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575070.171728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575070.619829:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575071.195729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575071.643812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575072.219721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575072.667802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575073.243696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575073.691808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575074.267713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575074.267725:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575074.267730:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575074.267731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575074.267765:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11827 00000100:00080000:1.0:1788575074.267777:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575074.267785:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575074.267844:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569918336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575074.267883:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575074.267888:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575074.267891:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575074.267897:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11827 00000100:00080000:1.0:1788575074.267901:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575074.267907:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575074.267919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575074.267926:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569918336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575074.267937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569918464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575074.267949:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575074.267954:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575074.267956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575074.267961:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11827 00000100:00080000:1.0:1788575074.267965:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575074.267970:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575074.267975:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575074.267990:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569918592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575074.268002:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575074.268005:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575074.268008:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575074.268012:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11827 00000100:00080000:1.0:1788575074.268017:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575074.268021:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575074.268027:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575074.268043:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569918720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575074.268054:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575074.268058:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575074.268060:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575074.268064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918464:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788575074.268068:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11827 00000100:00080000:1.0:1788575074.268076:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575074.268081:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575074.268088:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575074.268103:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569918848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575074.268109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575074.268145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575074.268174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575074.715777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575075.291782:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575075.739758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575076.315748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575076.763750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575077.339861:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575077.339873:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575077.339914:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575077.339929:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575077.339966:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575077.339975:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575077.339984:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575077.339990:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575077.340741:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575077] req@ffff920a52890380 x1875444569918848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575077.340775:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569918848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575077.340797:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575077.340803:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575077.340807:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918848:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575077.340821:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575077] req@ffff920a52891f80 x1875444569918464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575077.340832:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569918464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575077.340838:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575077.340842:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575077.340845:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575077.340852:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575077] req@ffff920a52891880 x1875444569918592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575077.340863:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569918592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575077.340870:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575077.340873:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575077.340876:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575077.340883:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575077] req@ffff920a52893480 x1875444569918720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575077.340893:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569918720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575077.340900:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575077.340904:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575077.340907:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569918720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575077.340914:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575074/real 1788575077] req@ffff920a52891500 x1875444569918336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575077.340925:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569918336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575077.340931:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575077.340935:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575077.340937:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569918336:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575077.787772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575078.363742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575078.811770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575079.387717:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575079.387727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575079.387728:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575079.387733:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575079.387744:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11832 00000100:00080000:0.0:1788575079.387753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575079.387760:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575079.387790:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569918976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575079.387819:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575079.387850:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575079.387853:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575079.387858:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11832 00000100:00080000:0.0:1788575079.387863:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575079.387867:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569918976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575079.387871:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575079.387887:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575079.387903:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569919104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575079.387914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575079.387917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575079.387920:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575079.387923:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11832 00000100:00080000:0.0:1788575079.387928:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575079.387932:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575079.387937:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788575079.387944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575079.387949:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569919232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575079.387960:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575079.387963:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575079.387965:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575079.387970:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11832 00000100:00080000:0.0:1788575079.387975:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575079.387980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff920a48350a80 x1875444569918976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575079.387981:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575079.387990:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575079.388002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569918976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575079.388006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569919360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575079.388027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575079.388028:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575079.388033:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575079.388035:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575079.388044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575079.388045:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11832 00000100:00080000:0.0:1788575079.388052:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575079.388054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569918976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575079.388060:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575079.388066:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575079.388078:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569919488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575079.388083:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575079.388115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575079.388130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff920a48353800 x1875444569919104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575079.388140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569919104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575079.388149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575079.388152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575079.388154:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575079.388160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575079.388186:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575079.388197:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff920a48350380 x1875444569919232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575079.388210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569919232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575079.388219:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575079.388222:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575079.388225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575079.388232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575079.388252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575079.388262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff920a48350700 x1875444569919360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575079.388274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569919360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575079.388282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575079.388286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575079.388289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575079.388306:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575079.388324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575079.388333:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575079/real 1788575079] req@ffff920a48353480 x1875444569919488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575079.388358:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569919488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575079.388373:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575079.388376:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575079.388380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575079.835734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575080.411865:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575080.859765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575081.435711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575081.883796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575082.459714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575082.907813:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575083.483698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575083.931812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575084.507732:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575084.507741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575084.507744:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575084.507750:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575084.507761:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11837 00000100:00080000:0.0:1788575084.507771:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575084.507779:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575084.507812:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569919616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575084.507841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575084.507845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575084.507846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575084.507852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11837 00000100:00080000:0.0:1788575084.507858:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575084.507863:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575084.507875:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575084.507889:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569919616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575084.507891:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569919744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575084.507928:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575084.507931:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575084.507933:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575084.507939:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11837 00000100:00080000:0.0:1788575084.507943:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575084.507947:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575084.507954:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575084.507968:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569919872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575084.507978:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575084.507981:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575084.507983:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575084.507994:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11837 00000800:00000100:2.0:1788575084.507994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575084.508002:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575084.508011:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575084.508018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575084.508034:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569920000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575084.508045:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575084.508047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff920a48351c00 x1875444569919616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.508057:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575084.508063:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575084.508072:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11837 00000100:00100000:2.0:1788575084.508077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569919616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575084.508078:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575084.508093:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575084.508102:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575084.508103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575084.508115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575084.508123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569919616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575084.508124:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569920128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575084.508165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575084.508188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575084.508198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff920a48351500 x1875444569919744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575084.508207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569919744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575084.508213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575084.508216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575084.508218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575084.508223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575084.508234:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575084.508240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff920a48353b80 x1875444569919872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575084.508248:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569919872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575084.508254:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575084.508256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575084.508257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569919872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575084.508261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575084.508272:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575084.508278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff920a48353480 x1875444569920000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575084.508285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569920000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575084.508291:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575084.508292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575084.508294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575084.508298:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575084.508308:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575084.508317:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575084/real 1788575084] req@ffff920a48350700 x1875444569920128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575084.508330:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569920128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575084.508339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575084.508342:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575084.508346:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575084.955751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575085.531706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575085.979726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575086.555689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575087.003773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575087.579739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575088.027836:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575088.603719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575089.051804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575089.627702:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575089.627710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575089.627713:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575089.627719:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575089.627734:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11843 00000100:00080000:2.0:1788575089.627756:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575089.627766:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575089.627816:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569920256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575089.627841:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575089.627844:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575089.627846:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575089.627849:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11843 00000100:00080000:2.0:1788575089.627852:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575089.627856:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575089.627872:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575089.627881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569920256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575089.627888:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569920384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575089.627901:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575089.627903:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575089.627905:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575089.627908:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11843 00000100:00080000:2.0:1788575089.627910:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575089.627912:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575089.627916:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575089.627925:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569920512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575089.627931:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575089.627933:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575089.627934:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575089.627936:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11843 00000100:00080000:2.0:1788575089.627938:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575089.627940:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575089.627943:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575089.627949:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569920640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575089.627954:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575089.627956:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575089.627957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575089.627959:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11843 00000100:00080000:2.0:1788575089.627961:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575089.627964:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575089.627966:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575089.627972:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569920768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575089.628026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575089.628061:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575089/real 1788575089] req@ffff920a60ac5180 x1875444569920256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575089.628081:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569920256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575089.628089:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575089.628093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575089.628096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569920256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575089.628119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575089.628137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575089.628146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575089/real 1788575089] req@ffff920a60ac7100 x1875444569920384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575089.628153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569920384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575089.628159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575089.628161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575089.628164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575089.628167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575089.628179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575089.628184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575089/real 1788575089] req@ffff920a60ac6a00 x1875444569920512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575089.628192:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569920512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575089.628202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575089.628204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575089.628205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575089.628209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575089.628220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575089.628225:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575089/real 1788575089] req@ffff920a60ac4a80 x1875444569920640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575089.628232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569920640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575089.628237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575089.628240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575089.628242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575089.628245:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575089.628255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575089.628260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575089/real 1788575089] req@ffff920a60ac4700 x1875444569920768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575089.628267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569920768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575089.628272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575089.628274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575089.628277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569920768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575090.075800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575090.651693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575091.099754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575091.675732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575092.123792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575092.699711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575093.147726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575093.723723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575094.171795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575094.747715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575094.747743:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575094.747750:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575094.747757:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575094.747769:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11848 00000100:00080000:1.0:1788575094.747781:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575094.747792:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575094.747839:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569920896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575094.747870:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575094.747874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575094.747876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575094.747880:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11848 00000100:00080000:1.0:1788575094.747885:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575094.747890:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575094.747900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569920896:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788575094.747903:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575094.747920:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569921024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575094.747933:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575094.747937:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575094.747939:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575094.747945:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11848 00000100:00080000:1.0:1788575094.747956:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575094.747962:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575094.747968:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575094.747972:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575094.747984:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569921152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575094.747995:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575094.748000:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575094.748002:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575094.748007:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11848 00000100:00080000:1.0:1788575094.748012:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575094.748017:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575094/real 1788575094] req@ffff920a52890e00 x1875444569920896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575094.748032:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575094.748036:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569920896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575094.748047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1788575094.748048:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569921280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575094.748054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575094.748062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569920896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575094.748067:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575094.748072:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575094.748074:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575094.748089:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11848 00000100:00080000:1.0:1788575094.748094:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575094.748099:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921024:192.168.202.157@tcp:38:kworker.0 00000008:00100000:1.0:1788575094.748106:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575094.748122:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569921408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575094.748127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575094/real 1788575094] req@ffff920a52893800 x1875444569921024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575094.748150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569921024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575094.748161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575094.748164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575094.748166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575094.748172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575094.748184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575094/real 1788575094] req@ffff920a52892d80 x1875444569921152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575094.748198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569921152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575094.748203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575094.748206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575094.748208:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575094.748212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575094.748231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575094/real 1788575094] req@ffff920a52892300 x1875444569921280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575094.748277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569921280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575094.748283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575094.748285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575094.748287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575094.748307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575094.748324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575094.748331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575094/real 1788575094] req@ffff920a52891180 x1875444569921408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575094.748338:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569921408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575094.748344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575094.748346:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575094.748348:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575095.195790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575095.771710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575096.219790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575096.795710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575097.243793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575097.819722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575098.267733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575098.843742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575099.291776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575099.867697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575099.867715:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575099.867723:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575099.867727:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575099.867740:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11853 00000100:00080000:3.0:1788575099.867751:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575099.867760:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575099.867806:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569921536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575099.867837:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575099.867841:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575099.867842:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575099.867846:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11853 00000100:00080000:3.0:1788575099.867850:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575099.867854:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575099.867866:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575099.867874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569921536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575099.867882:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569921664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575099.867893:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575099.867896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575099.867898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575099.867902:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11853 00000100:00080000:3.0:1788575099.867905:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575099.867909:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575099.867914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575099.867924:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569921792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575099.867933:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575099.867937:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575099.867940:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575099.867943:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11853 00000100:00080000:3.0:1788575099.867954:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575099.867959:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788575099.867964:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788575099.867967:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575099.867981:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569921920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575099.867990:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575099.867993:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575099.867995:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575099.867999:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11853 00000100:00080000:3.0:1788575099.868003:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575099.868004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575099/real 1788575099] req@ffff920a529b3100 x1875444569921536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575099.868009:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575099.868018:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575099.868025:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569921536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575099.868031:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569922048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575099.868037:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575099.868044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575099.868048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569921536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575099.868074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575099.868090:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575099.868098:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575099/real 1788575099] req@ffff920a529b1c00 x1875444569921664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575099.868107:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569921664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575099.868112:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575099.868113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575099.868115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575099.868119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575099.868129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575099.868134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575099/real 1788575099] req@ffff920a529b2a00 x1875444569921792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575099.868141:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569921792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575099.868146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575099.868147:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575099.868149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575099.868152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575099.868161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575099.868182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575099/real 1788575099] req@ffff920a529b3480 x1875444569921920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575099.868190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569921920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575099.868200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575099.868202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575099.868204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569921920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575099.868207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575099.868226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575099.868236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575099/real 1788575099] req@ffff920a529b2d80 x1875444569922048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575099.868246:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569922048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575099.868253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575099.868256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575099.868259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575100.315809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575100.891733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575101.339768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575101.915675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575102.363767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575102.939717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575103.387779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575103.963730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575104.411767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575104.987662:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575104.987664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575104.987671:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575104.987675:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575104.987683:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11858 00000100:00080000:3.0:1788575104.987691:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575104.987696:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575104.987724:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569922176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575104.987748:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575104.987755:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575104.987757:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575104.987763:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11858 00000100:00080000:3.0:1788575104.987767:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575104.987771:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575104.987779:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569922176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575104.987781:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575104.987799:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569922304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575104.987810:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575104.987815:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575104.987816:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575104.987820:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11858 00000100:00080000:3.0:1788575104.987824:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575104.987828:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575104.987833:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1788575104.987842:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575104.987845:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569922432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575104.987855:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575104.987858:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575104.987860:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575104.987863:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11858 00000100:00080000:3.0:1788575104.987867:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575104.987872:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575104.987877:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575104.987886:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569922560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575104.987891:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575104/real 1788575104] req@ffff920a529b2d80 x1875444569922176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575104.987908:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575104.987911:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569922176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575104.987913:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575104.987916:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575104.987924:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11858 00000100:00080000:3.0:1788575104.987930:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575104.987932:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575104.987933:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575104.987942:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:3.0:1788575104.987942:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575104.987949:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569922176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575104.987957:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569922688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575104.987976:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575104.988001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575104.988013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575104/real 1788575104] req@ffff920a529b3480 x1875444569922304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575104.988025:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569922304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575104.988033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575104.988036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575104.988039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575104.988045:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575104.988062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575104.988072:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575104/real 1788575104] req@ffff920a529b2a00 x1875444569922432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575104.988082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569922432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575104.988090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575104.988094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575104.988097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575104.988103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575104.988121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575104.988131:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575104/real 1788575104] req@ffff920a529b1c00 x1875444569922560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575104.988142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569922560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575104.988149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575104.988151:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575104.988154:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575104.988169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575104.988197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575104.988208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575104/real 1788575104] req@ffff920a529b3100 x1875444569922688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575104.988218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569922688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575129 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575104.988225:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575104.988227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575104.988230:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575105.435776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575106.011697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575106.459744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575107.035685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575107.483762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575108.059703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575108.507759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575109.083767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575109.531773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575110.107681:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575110.107685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575110.107690:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575110.107695:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.107706:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11863 00000100:00080000:3.0:1788575110.107716:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.107724:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575110.107762:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569922816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.107786:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.107789:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.107790:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.107794:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11863 00000100:00080000:3.0:1788575110.107796:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.107799:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575110.107808:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575110.107818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569922944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575110.107822:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569922816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575110.107829:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.107832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575110.107833:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.107836:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11863 00000100:00080000:3.0:1788575110.107838:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.107840:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575110.107842:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575110.107849:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569923072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.107855:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.107857:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575110.107858:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.107861:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11863 00000100:00080000:3.0:1788575110.107863:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.107865:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575110.107869:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575110.107875:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569923200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575110.107881:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575110.107882:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575110.107883:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575110.107885:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11863 00000100:00080000:3.0:1788575110.107888:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575110.107891:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575110.107895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575110.107896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575110.107903:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569923328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575110.107949:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff920a529b1f80 x1875444569922816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575110.107970:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569922816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575110.107990:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575110.107996:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575110.108000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569922816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575110.108022:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575110.108043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575110.108054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff920a529b0000 x1875444569922944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575110.108065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569922944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575110.108074:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575110.108077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575110.108080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569922944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575110.108086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575110.108102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575110.108109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff920a529b0700 x1875444569923072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575110.108117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569923072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575110.108121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575110.108123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575110.108125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575110.108128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575110.108137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575110.108142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff920a529b2300 x1875444569923200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575110.108148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569923200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575110.108153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575110.108154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575110.108156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575110.108158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575110.108167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575110.108171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575110/real 1788575110] req@ffff920a529b0380 x1875444569923328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575110.108177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569923328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575110.108185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575110.108187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575110.108189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575110.555774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575111.131722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575111.579761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575112.155720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575112.603768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575113.179734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575113.627779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575114.203687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575114.651753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575115.227685:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575115.227688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575115.227694:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575115.227699:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.227707:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11868 00000100:00080000:3.0:1788575115.227715:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.227720:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575115.227749:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569923456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.227770:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.227772:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575115.227774:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.227776:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11868 00000100:00080000:3.0:1788575115.227779:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.227781:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575115.227789:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575115.227798:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569923584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.227804:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.227806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575115.227808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.227812:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11868 00000100:00100000:0.0:1788575115.227813:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569923456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575115.227814:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.227819:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575115.227823:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575115.227830:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569923712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.227835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.227838:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575115.227839:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.227842:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11868 00000100:00080000:3.0:1788575115.227844:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.227847:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575115.227851:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575115.227856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569923840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575115.227866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575115.227868:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1788575115.227872:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575115.227872:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575115.227876:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11868 00000100:00080000:3.0:1788575115.227879:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575115.227881:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575115.227884:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575115.227892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569923968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575115.227905:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff920a529b3b80 x1875444569923456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575115.227917:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569923456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575115.227924:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575115.227927:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575115.227930:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569923456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575115.227935:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575115.227948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575115.227955:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff920a529b1180 x1875444569923584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575115.227961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569923584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575115.227970:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575115.227972:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575115.227974:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575115.227998:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575115.228009:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575115.228015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff920a529b1500 x1875444569923712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575115.228021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569923712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575115.228026:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575115.228028:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575115.228030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575115.228033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575115.228042:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575115.228047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff920a529b0e00 x1875444569923840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575115.228053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569923840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575115.228057:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575115.228058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575115.228061:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575115.228064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575115.228072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575115.228077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575115/real 1788575115] req@ffff920a529b0380 x1875444569923968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575115.228087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569923968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575115.228094:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575115.228097:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575115.228100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569923968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575115.675749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575116.251730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575116.699772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575117.275679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575117.723711:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575118.299689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575118.747754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575119.323659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575119.771751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575120.347708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575120.347713:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575120.347720:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575120.347724:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575120.347734:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11873 00000100:00080000:2.0:1788575120.347744:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575120.347750:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575120.347783:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569924096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575120.347806:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575120.347809:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575120.347810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575120.347813:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11873 00000100:00080000:2.0:1788575120.347815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575120.347817:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575120.347824:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575120.347831:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569924224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575120.347842:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575120.347844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575120.347845:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575120.347848:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11873 00000100:00080000:2.0:1788575120.347850:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575120.347851:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575120.347854:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575120.347858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569924096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575120.347862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569924352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575120.347871:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575120.347873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575120.347874:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575120.347876:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11873 00000100:00080000:2.0:1788575120.347877:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575120.347879:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575120.347882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575120.347888:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569924480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575120.347894:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575120.347895:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575120.347896:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575120.347898:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11873 00000100:00080000:2.0:1788575120.347901:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575120.347903:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575120.347906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575120.347912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569924608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575120.347925:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575120.347959:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff920a60ac6300 x1875444569924096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575120.347977:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569924096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575120.347989:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575120.347992:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575120.347996:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569924096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575120.348003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575120.348026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575120.348039:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff920a60ac5500 x1875444569924224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575120.348051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569924224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575120.348059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575120.348062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575120.348066:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575120.348088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575120.348109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575120.348119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff920a60ac4000 x1875444569924352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575120.348131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569924352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575120.348139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575120.348142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575120.348146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575120.348152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575120.348180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575120.348192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff920a60ac5c00 x1875444569924480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575120.348202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569924480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575120.348210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575120.348213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575120.348216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575120.348222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575120.348242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575120.348251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575120/real 1788575120] req@ffff920a60ac4380 x1875444569924608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575120.348263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569924608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575120.348268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575120.348270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575120.348272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575120.795759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575121.371695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575121.819767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575122.395680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575122.843720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575123.419699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575123.867831:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575124.443703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575124.891775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575125.467715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575125.467719:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575125.467728:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575125.467732:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575125.467742:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11878 00000100:00080000:1.0:1788575125.467753:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575125.467761:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575125.467802:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569924736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575125.467826:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575125.467829:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575125.467831:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575125.467835:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11878 00000100:00080000:1.0:1788575125.467839:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575125.467845:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575125.467853:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575125.467865:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569924864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575125.467866:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569924736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575125.467879:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575125.467882:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575125.467883:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575125.467887:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11878 00000100:00080000:1.0:1788575125.467890:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575125.467892:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575125.467895:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575125.467904:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569924992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575125.467910:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575125.467913:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575125.467914:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575125.467917:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11878 00000100:00080000:1.0:1788575125.467919:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575125.467921:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788575125.467924:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788575125.467925:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575125.467934:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569925120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575125.467940:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575125.467942:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575125.467943:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575125.467946:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11878 00000100:00080000:1.0:1788575125.467948:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575125.467950:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575125.467953:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575125.467955:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575125] req@ffff920a52893100 x1875444569924736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575125.467961:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569925248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575125.467973:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569924736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575125.467982:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575125.467985:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575125.467988:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569924736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575125.468005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575125.468020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575125.468027:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575125] req@ffff920a52890700 x1875444569924864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575125.468034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569924864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575125.468039:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575125.468041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575125.468043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575125.468046:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575125.468055:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575125.468060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575125] req@ffff920a52891180 x1875444569924992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575125.468066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569924992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575125.468071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575125.468072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575125.468074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569924992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575125.468077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575125.468087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575125.468092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575125] req@ffff920a52892300 x1875444569925120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575125.468099:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569925120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575125.468103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575125.468104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575125.468106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575125.468109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575125.468117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575125.468122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575125/real 1788575125] req@ffff920a52892d80 x1875444569925248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575125.468128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569925248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575125.468133:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575125.468134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575125.468136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575125.915738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575126.491823:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575126.939765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575127.515752:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575127.963783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575128.539719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575128.987785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575129.563692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575130.011758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575130.587719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575130.587729:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575130.587736:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575130.587740:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575130.587749:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11883 00000100:00080000:2.0:1788575130.587760:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575130.587767:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575130.587808:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569925376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575130.587839:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575130.587843:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575130.587845:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575130.587850:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11883 00000100:00080000:2.0:1788575130.587854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575130.587858:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575130.587867:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575130.587880:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569925504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575130.587889:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575130.587893:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575130.587895:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575130.587900:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11883 00000100:00080000:2.0:1788575130.587904:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575130.587908:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575130.587913:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575130.587929:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569925632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575130.587939:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575130.587943:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575130.587944:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575130.587949:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11883 00000100:00080000:2.0:1788575130.587952:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575130.587957:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575130.587962:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575130.587975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569925760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575130.587985:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575130.587988:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575130.587990:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575130.587993:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11883 00000100:00080000:2.0:1788575130.587997:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575130.588001:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575130.588007:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575130.588017:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569925888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575130.588028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569925376:192.168.202.157@tcp:250:kworker.0 00000800:00000100:1.0:1788575130.588077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575130.588100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff920a60ac4e00 x1875444569925376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.588111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569925376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.588117:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.588121:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575130.588123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569925376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575130.588127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575130.588150:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575130.588158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff920a60ac7800 x1875444569925504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.588165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569925504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.588170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.588172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575130.588174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575130.588177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575130.588186:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575130.588191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff920a60ac4380 x1875444569925632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.588198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569925632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.588207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.588209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575130.588211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575130.588213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575130.588223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575130.588227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff920a60ac5c00 x1875444569925760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.588233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569925760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.588238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.588240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575130.588242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575130.588259:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575130.588269:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575130.588274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575130/real 1788575130] req@ffff920a60ac4000 x1875444569925888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575130.588280:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569925888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575130.588284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575130.588286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575130.588289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569925888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575131.035782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575131.611718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575132.059742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575132.635697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575133.083774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575133.659662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575134.107758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575134.683704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575135.131716:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575135.707717:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575135.707718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575135.707729:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575135.707735:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575135.707747:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11889 00000100:00080000:0.0:1788575135.707758:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575135.707779:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575135.707824:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569926016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575135.707855:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575135.707858:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575135.707859:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575135.707863:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11889 00000100:00080000:0.0:1788575135.707866:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575135.707869:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575135.707880:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575135.707891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569926016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575135.707892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569926144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575135.707904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575135.707906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575135.707908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575135.707911:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11889 00000100:00080000:0.0:1788575135.707913:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575135.707915:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575135.707917:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575135.707924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569926272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575135.707933:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575135.707936:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575135.707938:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575135.707942:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11889 00000100:00080000:0.0:1788575135.707954:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575135.707960:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575135.707965:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575135.707973:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575135.707980:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569926400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575135.707986:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575135.707989:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575135.707991:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575135.707994:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11889 00000100:00080000:0.0:1788575135.707996:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575135.707999:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575135.708003:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575135.708010:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569926528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575135.708020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575135/real 1788575135] req@ffff920a48350380 x1875444569926016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575135.708040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569926016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575135.708051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575135.708055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575135.708058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569926016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575135.708084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575135.708107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575135.708119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575135/real 1788575135] req@ffff920a48353800 x1875444569926144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575135.708131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569926144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575135.708139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575135.708142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575135.708146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575135.708152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575135.708168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575135.708179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575135/real 1788575135] req@ffff920a48350a80 x1875444569926272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575135.708190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569926272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575135.708198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575135.708201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575135.708204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575135.708210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575135.708229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575135.708239:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575135/real 1788575135] req@ffff920a48350700 x1875444569926400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575135.708251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569926400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575135.708258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575135.708262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575135.708265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575135.708271:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575135.708289:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575135.708300:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575135/real 1788575135] req@ffff920a48353480 x1875444569926528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575135.708311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569926528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575135.708319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575135.708322:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575135.708325:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575136.155797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575136.731735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575137.179785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575137.755716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575138.203755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575138.779695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575139.227781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575139.803891:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575140.251790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575140.827670:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575140.827677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575140.827684:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575140.827690:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575140.827703:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11894 00000100:00080000:0.0:1788575140.827714:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575140.827723:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575140.827769:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569926656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575140.827801:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575140.827806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575140.827808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575140.827812:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11894 00000100:00080000:0.0:1788575140.827815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575140.827821:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575140.827833:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575140.827840:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569926656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575140.827848:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569926784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575140.827859:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575140.827863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575140.827865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575140.827869:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11894 00000100:00080000:0.0:1788575140.827873:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575140.827877:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575140.827882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575140.827893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569926912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575140.827903:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575140.827907:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575140.827910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575140.827913:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11894 00000100:00080000:0.0:1788575140.827917:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575140.827921:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575140.827926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575140.827939:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569927040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575140.827949:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575140.827953:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575140.827956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575140.827959:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926784:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575140.827964:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11894 00000100:00080000:0.0:1788575140.827970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575140.827975:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575140.827980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575140.827993:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569927168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575140.827995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575140.828019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575140.828038:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575141.275768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575141.851675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575142.299761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575142.875692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575143.323763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575143.899710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788575143.899859:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788575143.899899:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575143.899917:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575143.899961:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575143.899975:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575143.899986:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575143.899996:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788575143.900004:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575140/real 1788575143] req@ffff920a48351500 x1875444569926784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575143.900037:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569926784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575143.900052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575143.900058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575143.900072:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575143.900080:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575140/real 1788575143] req@ffff920a48351c00 x1875444569926912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575143.900092:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569926912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575143.900099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575143.900103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575143.900105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569926912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575143.900113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575140/real 1788575143] req@ffff920a48353480 x1875444569927040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575143.900123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569927040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575143.900130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575143.900134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575143.900137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575143.900143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575140/real 1788575143] req@ffff920a48353b80 x1875444569926656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575143.900155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569926656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575143.900163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575143.900167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575143.900170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569926656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575143.900201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575140/real 1788575143] req@ffff920a48350700 x1875444569927168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575143.900215:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569927168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575143.900223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575143.900226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575143.900229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575144.347773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575144.923727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575145.371739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575145.947672:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575145.947672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575145.947681:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575145.947685:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575145.947695:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11899 00000100:00080000:0.0:1788575145.947705:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575145.947712:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575145.947745:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569927296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575145.947771:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575145.947774:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575145.947776:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575145.947781:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11899 00000100:00080000:0.0:1788575145.947787:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575145.947791:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575145.947796:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569927296:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575145.947803:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575145.947820:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569927424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575145.947831:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575145.947835:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575145.947837:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575145.947841:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11899 00000100:00080000:0.0:1788575145.947846:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575145.947850:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575145.947854:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575145.947860:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575145.947873:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569927552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575145.947884:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575145.947885:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575145/real 1788575145] req@ffff920a48350a80 x1875444569927296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575145.947891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575145.947894:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575145.947903:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11899 00000100:00080000:0.0:1788575145.947907:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575145.947912:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575145.947915:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569927296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575145.947918:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575145.947932:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575145.947936:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569927680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575145.947940:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575145.947947:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569927296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575145.947967:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575145.947970:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575145.947972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575145.947977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11899 00000100:00080000:0.0:1788575145.947981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575145.947982:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927424:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575145.947987:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575145.947993:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575145.948007:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569927808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575145.948015:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575145.948032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575145/real 1788575145] req@ffff920a48353800 x1875444569927424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575145.948045:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569927424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575145.948053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575145.948056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575145.948059:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575145.948064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575145.948082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575145.948092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575145/real 1788575145] req@ffff920a48350380 x1875444569927552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575145.948101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569927552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575145.948108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575145.948110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575145.948113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575145.948118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575145.948135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575145.948144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575145/real 1788575145] req@ffff920a48350700 x1875444569927680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575145.948155:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569927680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575145.948162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575145.948165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575145.948168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575145.948182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575145.948203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575145.948214:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575145/real 1788575145] req@ffff920a48353b80 x1875444569927808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575145.948226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569927808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575170 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575145.948262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575145.948266:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575145.948269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569927808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575146.395753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575146.971664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575147.419725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575147.995671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575148.443751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575149.019669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575149.467757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575150.043704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575150.491803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575151.067662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575151.067706:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.067713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575151.067716:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.067724:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11904 00000100:00080000:2.0:1788575151.067732:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.067738:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575151.067769:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569927936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.067790:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.067793:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575151.067795:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.067798:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11904 00000100:00080000:2.0:1788575151.067801:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.067804:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575151.067814:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575151.067820:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569927936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575151.067827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569928064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.067835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.067838:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575151.067839:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.067842:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11904 00000100:00080000:2.0:1788575151.067844:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.067846:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575151.067849:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575151.067859:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569928192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.067866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.067868:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575151.067869:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.067871:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11904 00000100:00080000:2.0:1788575151.067877:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.067879:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575151.067882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575151.067890:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575151.067892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569928320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575151.067900:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575151.067903:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575151.067905:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575151.067908:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11904 00000100:00080000:2.0:1788575151.067917:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575151.067932:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575151.067937:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575151.067941:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff920a60ac5500 x1875444569927936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575151.067950:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569928448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575151.067963:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569927936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.067974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.067977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575151.067980:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569927936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575151.067999:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.068018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575151.068027:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff920a60ac6300 x1875444569928064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.068035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569928064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.068041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.068043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575151.068045:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575151.068049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575151.068076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575151.068096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff920a60ac4700 x1875444569928192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.068110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569928192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.068126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.068129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575151.068133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575151.068140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575151.068163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575151.068176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff920a60ac4a80 x1875444569928320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.068189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569928320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.068198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.068202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575151.068206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575151.068213:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575151.068236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575151.068248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575151/real 1788575151] req@ffff920a60ac6a00 x1875444569928448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575151.068256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569928448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575151.068262:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575151.068264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575151.068267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575151.515722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575152.091699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575152.539750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575153.115645:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575153.563757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575154.139797:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575154.587758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575155.163653:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575155.611746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575156.187690:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575156.187690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575156.187701:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575156.187704:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575156.187712:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11909 00000100:00080000:2.0:1788575156.187722:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575156.187730:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575156.187762:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569928576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575156.187783:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575156.187786:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575156.187787:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575156.187790:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11909 00000100:00080000:2.0:1788575156.187793:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575156.187796:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575156.187810:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575156.187815:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569928576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575156.187823:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569928704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575156.187836:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575156.187839:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575156.187840:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575156.187843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11909 00000100:00080000:2.0:1788575156.187846:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575156.187848:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575156.187851:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575156.187861:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569928832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575156.187867:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575156.187870:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575156.187871:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575156.187875:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575156.187876:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11909 00000100:00080000:2.0:1788575156.187881:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575156.187883:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575156.187888:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575156.187902:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569928960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575156.187913:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575156.187914:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff920a60ac7100 x1875444569928576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575156.187922:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575156.187927:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575156.187934:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11909 00000100:00080000:2.0:1788575156.187939:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575156.187941:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569928576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575156.187945:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575156.187952:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575156.187963:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788575156.187967:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569929088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575156.187972:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575156.187983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569928576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575156.188019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.188047:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575156.188060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff920a60ac5180 x1875444569928704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.188070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569928704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.188079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.188082:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575156.188087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575156.188094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575156.188124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575156.188136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff920a60ac6a00 x1875444569928832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.188148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569928832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.188157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.188160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575156.188165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575156.188172:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575156.188193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575156.188202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff920a60ac4a80 x1875444569928960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.188209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569928960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.188215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.188217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575156.188219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569928960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575156.188223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575156.188234:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575156.188241:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575156/real 1788575156] req@ffff920a60ac4700 x1875444569929088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575156.188248:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569929088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575156.188253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575156.188255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575156.188257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575156.635739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575157.211692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575157.659781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575158.235683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575158.683774:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575159.259684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575159.707773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575160.283680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575160.731818:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575161.307740:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575161.307746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575161.307756:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575161.307763:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.307776:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11914 00000100:00080000:1.0:1788575161.307791:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.307801:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575161.307848:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569929216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.307879:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.307882:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575161.307884:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.307887:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11914 00000100:00080000:1.0:1788575161.307890:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.307894:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575161.307905:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575161.307915:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569929344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.307924:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.307927:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575161.307928:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.307932:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11914 00000100:00080000:1.0:1788575161.307936:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575161.307937:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569929216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575161.307940:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575161.307949:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575161.307963:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569929472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.307973:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.307977:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575161.307979:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.307983:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11914 00000100:00080000:1.0:1788575161.307987:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.307992:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575161.307998:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575161.308010:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569929600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575161.308017:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575161.308020:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575161.308022:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575161.308025:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11914 00000100:00080000:1.0:1788575161.308028:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575161.308030:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575161.308033:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575161.308040:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575161.308043:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569929728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575161.308088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff920a52893800 x1875444569929216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.308111:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569929216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.308123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.308128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575161.308132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569929216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575161.308160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575161.308185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575161.308199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff920a52890e00 x1875444569929344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.308212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569929344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.308222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.308225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575161.308229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575161.308236:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575161.308254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575161.308265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff920a52891500 x1875444569929472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.308277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569929472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.308286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.308290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575161.308293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575161.308315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575161.308337:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575161.308358:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff920a52893480 x1875444569929600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.308371:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569929600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.308386:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.308391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575161.308394:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575161.308401:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575161.308423:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575161.308433:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575161/real 1788575161] req@ffff920a52891880 x1875444569929728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575161.308444:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569929728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575161.308452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575161.308455:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575161.308458:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575161.755798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575162.331751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575162.779775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575163.355722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575163.803805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575164.379743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575164.827772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575165.403743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575165.851762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575166.427704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575166.427717:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575166.427725:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575166.427729:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575166.427740:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11919 00000100:00080000:2.0:1788575166.427750:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575166.427758:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575166.427790:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569929856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575166.427818:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575166.427823:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575166.427825:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575166.427830:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11919 00000100:00080000:2.0:1788575166.427833:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575166.427838:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575166.427849:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575166.427857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569929856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575166.427864:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569929984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575166.427877:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575166.427882:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575166.427885:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575166.427890:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11919 00000100:00080000:2.0:1788575166.427895:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575166.427899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575166.427905:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575166.427916:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569930112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575166.427926:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575166.427930:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575166.427935:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575166.427937:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575166.427943:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11919 00000100:00080000:2.0:1788575166.427947:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575166.427952:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575166.427957:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575166.427961:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff920a60ac6300 x1875444569929856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575166.427970:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569930240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575166.427982:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575166.427983:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569929856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575166.428005:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575166.428008:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575166.428010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575166.428014:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11919 00000100:00080000:0.0:1788575166.428016:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575166.428021:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575166.428022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569929856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575166.428030:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575166.428036:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575166.428047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575166.428049:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569930368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575166.428078:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575166.428093:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff920a60ac5500 x1875444569929984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575166.428103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569929984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575166.428109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575166.428112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575166.428115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569929984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575166.428119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575166.428141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575166.428148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff920a60ac4000 x1875444569930112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575166.428156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569930112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575166.428162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575166.428164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575166.428166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575166.428169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575166.428182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575166.428188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff920a60ac5c00 x1875444569930240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575166.428197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569930240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575166.428203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575166.428206:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575166.428210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575166.428227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575166.428250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575166.428266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575166/real 1788575166] req@ffff920a60ac4380 x1875444569930368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575166.428278:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569930368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575166.428293:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575166.428298:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575166.428302:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575166.875749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575167.451706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575167.899785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575168.475683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575168.923754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575169.499677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575169.947772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575170.523669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575170.971702:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575171.547679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575171.547708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575171.547717:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575171.547722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575171.547733:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11924 00000100:00080000:2.0:1788575171.547743:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575171.547752:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575171.547790:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569930496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575171.547817:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575171.547821:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575171.547823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575171.547828:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11924 00000100:00080000:2.0:1788575171.547831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575171.547835:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575171.547845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575171.547859:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569930624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575171.547870:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575171.547874:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575171.547875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575171.547880:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11924 00000100:00100000:0.0:1788575171.547887:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569930496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575171.547891:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575171.547900:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575171.547906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575171.547922:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569930752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575171.547933:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575171.547937:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575171.547939:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575171.547943:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11924 00000100:00080000:2.0:1788575171.547954:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575171.547959:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788575171.547965:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575171.547967:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575171.547982:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569930880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575171.547992:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575171.547995:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575171.547997:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575171.548005:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11924 00000100:00100000:0.0:1788575171.548006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff920a60ac7800 x1875444569930496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575171.548008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575171.548016:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575171.548025:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575171.548038:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569931008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575171.548042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569930496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575171.548105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575171.548110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575171.548115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569930496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575171.548139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575171.548167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575171.548182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff920a60ac4e00 x1875444569930624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575171.548196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569930624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575171.548206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575171.548210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575171.548214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575171.548221:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575171.548255:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575171.548268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff920a60ac4380 x1875444569930752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575171.548283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569930752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575171.548292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575171.548296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575171.548300:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575171.548307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575171.548330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575171.548343:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff920a60ac5c00 x1875444569930880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575171.548357:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569930880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575171.548376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575171.548380:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575171.548384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569930880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575171.548391:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575171.548414:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575171.548427:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575171/real 1788575171] req@ffff920a60ac4000 x1875444569931008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575171.548440:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569931008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575171.548449:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575171.548453:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575171.548456:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575171.995784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575172.571669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575173.019762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575173.595713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575174.043775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575174.619693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575175.067783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575175.643679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575176.091724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575176.667708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575176.667722:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575176.667730:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575176.667735:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575176.667744:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11930 00000100:00080000:1.0:1788575176.667753:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575176.667760:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575176.667788:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569931136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575176.667811:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575176.667814:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575176.667815:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575176.667818:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11930 00000100:00080000:1.0:1788575176.667820:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575176.667822:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575176.667829:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575176.667837:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569931264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575176.667843:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575176.667845:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575176.667846:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575176.667848:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11930 00000100:00080000:1.0:1788575176.667850:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575176.667851:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575176.667853:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575176.667858:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569931392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575176.667863:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575176.667865:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575176.667872:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575176.667876:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11930 00000100:00080000:1.0:1788575176.667878:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575176.667880:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575176.667883:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575176.667889:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569931520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575176.667896:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575176.667897:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569931136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575176.667898:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575176.667901:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575176.667907:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11930 00000100:00080000:1.0:1788575176.667909:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575176.667911:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575176.667913:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575176.667920:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569931648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575176.667957:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575176.667985:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575176/real 1788575176] req@ffff920a52891f80 x1875444569931136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575176.667998:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569931136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575176.668005:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575176.668008:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575176.668011:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569931136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575176.668015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575176.668028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575176.668035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575176/real 1788575176] req@ffff920a52890380 x1875444569931264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575176.668042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569931264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575176.668047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575176.668048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575176.668050:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575176.668053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575176.668062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575176.668067:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575176/real 1788575176] req@ffff920a52893b80 x1875444569931392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575176.668073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569931392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575176.668082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575176.668084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575176.668085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575176.668103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575176.668113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575176.668118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575176/real 1788575176] req@ffff920a52890000 x1875444569931520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575176.668124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569931520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575176.668128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575176.668129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575176.668131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575176.668134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575176.668142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575176.668147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575176/real 1788575176] req@ffff920a52892680 x1875444569931648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575176.668152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569931648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575176.668156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575176.668157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575176.668160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575177.115726:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575177.691654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575178.139696:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575178.715700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575179.163744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575179.739676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575180.187726:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575180.763686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575181.211739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575181.787679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575181.787691:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575181.787698:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575181.787703:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575181.787713:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11935 00000100:00080000:0.0:1788575181.787727:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575181.787734:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575181.787770:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569931776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575181.787797:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575181.787801:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575181.787803:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575181.787807:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11935 00000100:00080000:0.0:1788575181.787811:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575181.787815:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575181.787824:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575181.787838:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569931776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575181.787839:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569931904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575181.787854:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575181.787857:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575181.787859:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575181.787864:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11935 00000100:00080000:0.0:1788575181.787867:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575181.787872:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575181.787878:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575181.787890:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569932032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575181.787900:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575181.787903:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575181.787907:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575181.787908:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575181.787915:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11935 00000100:00080000:0.0:1788575181.787919:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575181.787924:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575181.787929:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575181.787942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569932160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575181.787947:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575181/real 1788575181] req@ffff920a48353480 x1875444569931776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575181.787957:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575181.787965:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575181.787969:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575181.787974:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11935 00000100:00080000:0.0:1788575181.787979:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575181.787981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569931776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575181.787985:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575181.787995:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575181.787997:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575181.788009:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569932288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575181.788010:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575181.788016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569931776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575181.788041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575181.788059:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575181.788088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575181/real 1788575181] req@ffff920a48351c00 x1875444569931904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575181.788097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569931904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575181.788102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575181.788104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575181.788107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569931904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575181.788111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575181.788124:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575181.788130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575181/real 1788575181] req@ffff920a48351500 x1875444569932032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575181.788140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569932032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575181.788153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575181.788156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575181.788159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575181.788165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575181.788184:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575181.788194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575181/real 1788575181] req@ffff920a48353b80 x1875444569932160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575181.788206:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569932160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575181.788213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575181.788216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575181.788217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575181.788222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575181.788235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575181.788242:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575181/real 1788575181] req@ffff920a48350700 x1875444569932288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575181.788249:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569932288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575181.788253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575181.788255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575181.788257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575182.235736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575182.811677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575183.259767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575183.835655:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575184.283731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575184.859674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575185.307746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575185.883668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575186.331737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575186.907685:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575186.907691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575186.907693:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575186.907700:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575186.907710:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11940 00000100:00080000:2.0:1788575186.907720:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575186.907726:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575186.907789:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569932416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575186.907817:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575186.907821:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575186.907823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575186.907827:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11940 00000100:00080000:2.0:1788575186.907831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575186.907835:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575186.907845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575186.907855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569932416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575186.907861:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569932544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575186.907872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575186.907876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575186.907878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575186.907882:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11940 00000100:00080000:2.0:1788575186.907885:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575186.907889:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575186.907894:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575186.907913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575186.907914:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569932672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575186.907925:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575186.907928:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575186.907930:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575186.907935:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11940 00000100:00080000:2.0:1788575186.907938:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575186.907944:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575186/real 1788575186] req@ffff920a60ac5500 x1875444569932416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575186.907945:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575186.907954:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575186.907961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569932416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575186.907969:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569932800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575186.907972:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575186.907978:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575186.907984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569932416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575186.907988:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575186.907993:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575186.907995:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575186.907999:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11940 00000100:00080000:2.0:1788575186.908003:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575186.908005:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932544:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575186.908010:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575186.908017:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575186.908023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575186.908032:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569932928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575186.908035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575186/real 1788575186] req@ffff920a60ac6300 x1875444569932544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575186.908061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569932544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575186.908067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575186.908069:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575186.908072:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575186.908078:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575186.908088:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575186.908093:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575186/real 1788575186] req@ffff920a60ac4700 x1875444569932672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575186.908099:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569932672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575186.908104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575186.908106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575186.908107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575186.908110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575186.908120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575186.908125:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575186/real 1788575186] req@ffff920a60ac4a80 x1875444569932800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575186.908131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569932800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575186.908135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575186.908137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575186.908138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575186.908145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575186.908154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575186.908160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575186/real 1788575186] req@ffff920a60ac6a00 x1875444569932928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575186.908172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569932928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575186.908177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575186.908178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575186.908180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569932928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575187.355757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575187.931681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575188.379760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575188.955680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575189.403752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575189.979643:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575190.427720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575191.003671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575191.451784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575192.027671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575192.027680:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575192.027690:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575192.027695:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575192.027705:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11945 00000100:00080000:0.0:1788575192.027715:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575192.027723:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575192.027756:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569933056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575192.027774:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575192.027777:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575192.027779:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575192.027784:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11945 00000100:00080000:0.0:1788575192.027788:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575192.027792:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575192.027801:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569933056:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575192.027803:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575192.027820:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569933184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575192.027831:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575192.027835:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575192.027837:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575192.027840:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11945 00000100:00080000:0.0:1788575192.027844:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575192.027847:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575192.027851:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575192.027856:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575192.027861:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569933312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575192.027869:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575192.027871:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575192.027873:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575192.027876:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11945 00000100:00080000:0.0:1788575192.027878:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575192.027881:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575192.027884:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575192.027889:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575192] req@ffff920a48350380 x1875444569933056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575192.027892:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569933440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575192.027919:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575192.027922:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575192.027923:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575192.027926:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11945 00000100:00080000:0.0:1788575192.027928:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575192.027933:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575192.027934:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569933056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575192.027935:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575192.027945:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569933568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575192.027953:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575192.027960:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575192.027965:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569933056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575192.027988:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575192.028013:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575192.028028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575192] req@ffff920a48353800 x1875444569933184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575192.028041:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569933184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575192.028049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575192.028059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575192.028063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575192.028070:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575192.028091:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575192.028103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575192] req@ffff920a48350a80 x1875444569933312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575192.028115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569933312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575192.028122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575192.028126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575192.028129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575192.028135:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575192.028156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575192.028167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575192] req@ffff920a48350700 x1875444569933440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575192.028178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569933440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575192.028186:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575192.028189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575192.028192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575192.028197:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575192.028218:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575192.028230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575192/real 1788575192] req@ffff920a48353b80 x1875444569933568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575192.028241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569933568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575192.028248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575192.028252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575192.028255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575192.475740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575193.051679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575193.499729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575194.075688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575194.523798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575195.099684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575195.547735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575196.123654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575196.571832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575197.147668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575197.147680:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.147686:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575197.147690:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.147700:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11950 00000100:00080000:1.0:1788575197.147711:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.147717:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575197.147756:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569933696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.147781:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.147784:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575197.147785:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.147788:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11950 00000100:00080000:1.0:1788575197.147791:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.147794:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575197.147803:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575197.147812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569933696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575197.147814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569933824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.147825:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.147827:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575197.147828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.147831:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11950 00000100:00080000:1.0:1788575197.147832:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.147834:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575197.147837:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575197.147846:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569933952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.147852:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.147854:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575197.147856:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.147859:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11950 00000100:00080000:1.0:1788575197.147871:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.147874:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575197.147878:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575197.147886:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575197.147887:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569934080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575197.147896:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575197.147898:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575197.147900:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575197.147902:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11950 00000100:00080000:1.0:1788575197.147905:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575197.147908:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575197.147912:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575197.147923:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569934208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575197.147925:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff920a52892a00 x1875444569933696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.147963:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569933696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.147971:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.147975:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575197.147977:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569933696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575197.147996:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.148010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575197.148017:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff920a52890a80 x1875444569933824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.148024:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569933824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.148029:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.148030:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575197.148032:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575197.148035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575197.148045:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575197.148050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff920a52892680 x1875444569933952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.148057:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569933952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.148066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.148067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575197.148069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569933952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575197.148071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575197.148080:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575197.148085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff920a52890000 x1875444569934080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.148091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569934080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.148095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.148096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575197.148098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575197.148101:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575197.148110:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575197.148114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575197/real 1788575197] req@ffff920a52893b80 x1875444569934208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575197.148120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569934208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575197.148124:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575197.148125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575197.148128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575197.595736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575198.171699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575198.619757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575199.195641:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575199.643748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575200.219690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575200.667810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575201.243721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575201.691763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575202.267741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575202.267742:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575202.267757:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575202.267767:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.267779:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11955 00000100:00080000:0.0:1788575202.267790:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575202.267799:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575202.267833:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569934336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575202.267867:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575202.267870:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575202.267871:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.267875:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11955 00000100:00080000:0.0:1788575202.267878:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575202.267881:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575202.267890:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575202.267900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569934336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575202.267905:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569934464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575202.267926:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575202.267929:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575202.267933:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.267941:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11955 00000100:00080000:0.0:1788575202.267944:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575202.267948:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575202.267955:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575202.267965:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569934592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575202.267972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575202.267974:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575202.267975:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.267977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11955 00000100:00080000:0.0:1788575202.267981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575202.267981:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575202.267987:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575202.267992:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575202.268005:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569934720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575202.268012:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575202.268014:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575202.268015:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575202.268018:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11955 00000100:00080000:0.0:1788575202.268020:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575202.268023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff920a48351500 x1875444569934336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575202.268025:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575202.268030:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575202.268039:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569934848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575202.268045:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569934336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575202.268055:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575202.268058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575202.268061:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569934336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575202.268080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575202.268108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575202.268118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff920a48351c00 x1875444569934464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575202.268131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569934464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575202.268137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575202.268140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575202.268142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575202.268146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575202.268161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575202.268167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff920a48353480 x1875444569934592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575202.268178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569934592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575202.268182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575202.268184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575202.268187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575202.268190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575202.268217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575202.268228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff920a48353b80 x1875444569934720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575202.268236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569934720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575202.268248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575202.268251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575202.268253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575202.268263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575202.268280:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575202.268301:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575202/real 1788575202] req@ffff920a48350700 x1875444569934848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575202.268313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569934848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575202.268322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575202.268324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575202.268326:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569934848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575202.715772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575203.291700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575203.739741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575204.315694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575204.763796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575205.339707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575205.787791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575206.363716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575206.811820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575207.387705:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575207.387717:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575207.387722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575207.387733:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11960 00000100:00080000:0.0:1788575207.387743:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575207.387751:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575207.387797:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569934976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575207.387798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575207.387834:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575207.387839:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575207.387842:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575207.387847:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11960 00000100:00080000:0.0:1788575207.387851:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575207.387856:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575207.387865:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575207.387877:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569935104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575207.387885:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575207.387887:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575207.387889:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575207.387892:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11960 00000100:00080000:0.0:1788575207.387894:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575207.387896:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575207.387898:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575207.387905:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569935232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575207.387912:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575207.387915:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575207.387917:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575207.387922:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11960 00000100:00080000:0.0:1788575207.387926:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575207.387929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569934976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575207.387932:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575207.387942:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575207.387958:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569935360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575207.387970:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575207.387974:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575207.387976:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575207.387980:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11960 00000100:00080000:0.0:1788575207.387983:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575207.387987:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575207.387991:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575207.388002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569935488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575207.388043:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575207.388077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575207.388108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575207.388132:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575207.835735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575208.411723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575208.859767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575209.435678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575209.883706:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575210.459672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575210.459840:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575210.459875:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575210.459890:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575210.459928:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575210.459940:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575210.459953:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575210.459966:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575210.459971:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575210] req@ffff920a48350a80 x1875444569934976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575210.460000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569934976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575210.460011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575210.460015:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575210.460020:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569934976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575210.460028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575210] req@ffff920a48353800 x1875444569935104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575210.460040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569935104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575210.460053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575210.460057:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575210.460060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575210.460067:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575210] req@ffff920a48350380 x1875444569935232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575210.460078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569935232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575210.460086:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575210.460090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575210.460093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575210.460118:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575210] req@ffff920a48350700 x1875444569935360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575210.460131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569935360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575210.460139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575210.460143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575210.460146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575210.460160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575207/real 1788575210] req@ffff920a48353b80 x1875444569935488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575210.460173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569935488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575210.460182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575210.460186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575210.460190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575210.907752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575211.483670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575211.931756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575212.507741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575212.507759:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.507771:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575212.507777:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.507788:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11965 00000100:00080000:0.0:1788575212.507800:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.507808:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575212.507847:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569935616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575212.507878:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.507882:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575212.507884:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.507889:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11965 00000100:00080000:0.0:1788575212.507894:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.507900:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575212.507911:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575212.507913:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569935616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575212.507930:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569935744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575212.507943:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.507947:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575212.507949:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.507954:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11965 00000100:00080000:0.0:1788575212.507958:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.507963:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575212.507968:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575212.507981:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569935872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575212.507989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575212.507998:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.508003:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575212.508006:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575212.508011:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11965 00000100:00080000:0.0:1788575212.508016:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.508021:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575212.508027:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575212.508032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff920a48353480 x1875444569935616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575212.508044:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569936000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575212.508057:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569935616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575212.508066:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575212.508074:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575212.508077:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575212.508080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575212.508089:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575212.508090:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11965 00000100:00100000:2.0:1788575212.508096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569935616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575212.508098:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575212.508108:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575212.508114:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575212.508128:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569936128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575212.508139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575212.508171:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575212.508186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff920a48351c00 x1875444569935744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575212.508198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569935744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575212.508207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575212.508211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575212.508215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575212.508222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575212.508243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575212.508255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff920a48351500 x1875444569935872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575212.508267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569935872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575212.508276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575212.508279:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575212.508283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569935872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575212.508289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575212.508310:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575212.508322:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff920a48353b80 x1875444569936000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575212.508334:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569936000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575212.508344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575212.508348:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575212.508351:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575212.508370:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575212.508391:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575212.508403:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575212/real 1788575212] req@ffff920a48350700 x1875444569936128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575212.508415:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569936128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575212.508438:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575212.508441:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575212.508444:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575212.955802:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575213.531711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575213.979755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575214.555701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575215.003740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575215.579709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575216.027743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575216.603665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575217.051715:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575217.627733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575217.627735:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575217.627744:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575217.627750:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575217.627759:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11971 00000100:00080000:2.0:1788575217.627768:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575217.627776:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575217.627810:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569936256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575217.627835:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575217.627838:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575217.627839:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575217.627843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11971 00000100:00080000:2.0:1788575217.627845:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575217.627847:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575217.627856:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575217.627865:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569936384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575217.627879:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575217.627881:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575217.627882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575217.627885:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11971 00000100:00080000:2.0:1788575217.627887:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575217.627892:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575217.627896:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575217.627900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569936256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575217.627908:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569936512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575217.627916:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575217.627919:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575217.627920:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575217.627923:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11971 00000100:00080000:2.0:1788575217.627925:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575217.627927:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575217.627930:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575217.627941:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569936640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575217.627947:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575217.627949:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575217.627950:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575217.627953:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11971 00000100:00080000:2.0:1788575217.627956:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575217.627962:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575217.627966:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575217.627973:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569936768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575217.627994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575217.628047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575217/real 1788575217] req@ffff920a60ac5180 x1875444569936256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.628072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569936256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.628090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.628093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575217.628097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569936256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575217.628124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575217.628145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575217.628154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575217/real 1788575217] req@ffff920a60ac7100 x1875444569936384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.628163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569936384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.628168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.628171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575217.628173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575217.628177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575217.628189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575217.628196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575217/real 1788575217] req@ffff920a60ac6a00 x1875444569936512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.628203:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569936512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.628208:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.628225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575217.628227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575217.628233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575217.628250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575217.628258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575217/real 1788575217] req@ffff920a60ac4a80 x1875444569936640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.628266:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569936640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.628272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.628274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575217.628276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575217.628280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575217.628290:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575217.628296:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575217/real 1788575217] req@ffff920a60ac4700 x1875444569936768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575217.628303:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569936768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575217.628308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575217.628310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575217.628311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569936768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575218.075777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575218.651684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575219.099758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575219.675695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575220.123768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575220.699704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575221.147768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575221.723697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575222.171751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575222.747702:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575222.747705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575222.747715:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575222.747721:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575222.747735:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11976 00000100:00080000:2.0:1788575222.747749:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575222.747759:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575222.747805:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569936896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575222.747830:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575222.747833:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575222.747842:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575222.747845:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11976 00000100:00080000:2.0:1788575222.747849:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575222.747852:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575222.747863:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575222.747875:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569937024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575222.747884:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575222.747889:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575222.747891:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575222.747892:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569936896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575222.747898:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11976 00000100:00080000:2.0:1788575222.747905:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575222.747910:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575222.747915:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575222.747929:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569937152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575222.747940:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575222.747944:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575222.747946:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575222.747951:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11976 00000100:00080000:2.0:1788575222.747956:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575222.747960:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575222.747965:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575222.747975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569937280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575222.747984:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575222.747985:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575222.747990:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575222.747991:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575222.747994:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11976 00000100:00080000:2.0:1788575222.747997:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575222.748014:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575222.748019:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575222.748028:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569937408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575222.748062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575222/real 1788575222] req@ffff920a60ac6300 x1875444569936896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575222.748094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569936896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575222.748108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575222.748113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575222.748117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569936896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575222.748145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575222.748172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575222.748186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575222/real 1788575222] req@ffff920a60ac5500 x1875444569937024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575222.748201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569937024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575222.748210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575222.748214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575222.748218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575222.748225:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575222.748246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575222.748259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575222/real 1788575222] req@ffff920a60ac4000 x1875444569937152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575222.748273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569937152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575222.748289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575222.748293:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575222.748297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575222.748303:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575222.748324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575222.748336:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575222/real 1788575222] req@ffff920a60ac5c00 x1875444569937280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575222.748348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569937280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575222.748357:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575222.748361:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575222.748365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575222.748371:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575222.748391:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575222.748403:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575222/real 1788575222] req@ffff920a60ac4380 x1875444569937408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575222.748418:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569937408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575222.748426:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575222.748429:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575222.748433:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575223.195797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575223.771738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575224.219779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575224.795682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575225.243759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575225.819717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575226.267795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575226.843684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575227.291765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575227.867739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575227.867752:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575227.867758:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575227.867762:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575227.867770:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11981 00000100:00080000:0.0:1788575227.867779:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575227.867786:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575227.867831:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569937536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575227.867858:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575227.867861:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575227.867863:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575227.867866:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11981 00000100:00080000:0.0:1788575227.867869:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575227.867871:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575227.867879:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575227.867889:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569937664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575227.867897:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575227.867900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575227.867902:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575227.867907:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569937536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575227.867908:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11981 00000100:00080000:0.0:1788575227.867913:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575227.867916:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575227.867920:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575227.867930:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569937792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575227.867940:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575227.867944:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575227.867946:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575227.867950:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11981 00000100:00080000:0.0:1788575227.867955:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575227.867960:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788575227.867965:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575227.867969:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575227.867986:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569937920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575227.867997:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575227.867999:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575227/real 1788575227] req@ffff920a48350380 x1875444569937536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575227.868006:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575227.868012:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575227.868018:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11981 00000100:00100000:2.0:1788575227.868020:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569937536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575227.868024:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575227.868032:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788575227.868033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788575227.868037:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575227.868039:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575227.868044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569937536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575227.868053:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569938048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575227.868066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575227.868086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575227.868095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575227/real 1788575227] req@ffff920a48353800 x1875444569937664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575227.868103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569937664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575227.868109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575227.868111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575227.868113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575227.868116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575227.868128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575227.868134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575227/real 1788575227] req@ffff920a48350a80 x1875444569937792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575227.868142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569937792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575227.868147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575227.868149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575227.868151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575227.868154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575227.868165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575227.868170:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575227/real 1788575227] req@ffff920a48350700 x1875444569937920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575227.868177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569937920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575227.868183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575227.868184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575227.868186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569937920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575227.868196:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575227.868205:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575227.868211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575227/real 1788575227] req@ffff920a48353b80 x1875444569938048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575227.868222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569938048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575227.868228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575227.868229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575227.868231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575228.315808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575228.891661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575229.339761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575229.915699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575230.363742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575230.939723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575231.387776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575231.963729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575232.411747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575232.987673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575232.987674:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575232.987683:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575232.987687:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575232.987695:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11986 00000100:00080000:0.0:1788575232.987705:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575232.987712:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575232.987744:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569938176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575232.987766:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575232.987770:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575232.987772:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575232.987778:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11986 00000100:00080000:0.0:1788575232.987783:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575232.987788:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575232.987795:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569938176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575232.987799:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575232.987819:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569938304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575232.987830:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575232.987833:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575232.987835:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575232.987839:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11986 00000100:00080000:0.0:1788575232.987843:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575232.987845:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575232.987849:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788575232.987853:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575232.987863:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569938432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575232.987872:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575232.987874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575232.987876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575232.987879:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11986 00000100:00080000:0.0:1788575232.987882:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575232.987888:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575232.987891:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575232.987895:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575232/real 1788575232] req@ffff920a48351500 x1875444569938176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575232.987901:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569938560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575232.987920:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575232.987923:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569938176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575232.987926:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575232.987935:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575232.987937:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575232.987940:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575232.987948:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11986 00000100:00100000:2.0:1788575232.987948:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569938176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575232.987958:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575232.987964:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575232.987971:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938304:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788575232.987973:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575232.987992:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569938688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575232.987996:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575232.988009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575232/real 1788575232] req@ffff920a48351c00 x1875444569938304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575232.988026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569938304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575232.988032:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575232.988035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575232.988037:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575232.988041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575232.988052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575232.988058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575232/real 1788575232] req@ffff920a48353480 x1875444569938432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575232.988066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569938432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575232.988071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575232.988072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575232.988074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575232.988077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575232.988087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575232.988093:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575232/real 1788575232] req@ffff920a48353b80 x1875444569938560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575232.988100:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569938560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575232.988109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575232.988111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575232.988113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575232.988124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575232.988134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575232.988140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575232/real 1788575232] req@ffff920a48350700 x1875444569938688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575232.988147:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569938688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575257 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575232.988152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575232.988154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575232.988156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575233.435753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575234.011679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575234.459734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575235.035707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575235.483749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575236.059687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575236.507799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575237.083724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575237.531780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575238.107667:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575238.107677:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575238.107682:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575238.107683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575238.107695:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11991 00000100:00080000:0.0:1788575238.107708:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575238.107718:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575238.107771:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569938816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575238.107807:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575238.107811:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575238.107813:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575238.107820:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11991 00000100:00080000:0.0:1788575238.107827:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575238.107831:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575238.107845:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575238.107857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569938816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575238.107877:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569938944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575238.107890:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575238.107894:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575238.107895:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575238.107900:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11991 00000100:00080000:0.0:1788575238.107904:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575238.107909:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575238.107914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575238.107927:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569939072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575238.107936:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575238.107950:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575238.107956:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575238.107957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575238.107969:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11991 00000100:00080000:0.0:1788575238.107976:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575238.107986:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575238.107996:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575238.108012:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff920a48350a80 x1875444569938816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575238.108019:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569939200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575238.108048:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575238.108048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569938816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575238.108067:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575238.108075:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575238.108076:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575238.108086:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11991 00000100:00080000:0.0:1788575238.108091:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575238.108098:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575238.108104:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575238.108120:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569939328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575238.108142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575238.108146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569938816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575238.108183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575238.108204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575238.108345:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff920a48353800 x1875444569938944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575238.108360:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569938944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.108382:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575238.108385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575238.108387:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569938944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575238.108436:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575238.108465:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575238.108475:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff920a48350380 x1875444569939072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575238.108483:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569939072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.108493:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575238.108496:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575238.108498:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575238.108502:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575238.108514:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575238.108520:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff920a48350700 x1875444569939200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575238.108528:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569939200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.108533:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575238.108534:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575238.108536:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575238.108540:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575238.108550:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575238.108556:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575238/real 1788575238] req@ffff920a48353b80 x1875444569939328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575238.108562:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569939328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575238.108567:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575238.108569:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575238.108571:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575238.555761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575239.131705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575239.579760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575240.155688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575240.603762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575241.179692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575241.627788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575242.203717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575242.651815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575243.227713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575243.227714:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.227725:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575243.227729:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.227740:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 11996 00000100:00080000:1.0:1788575243.227751:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.227761:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575243.227813:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569939456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.227845:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.227850:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575243.227852:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.227857:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11996 00000100:00080000:1.0:1788575243.227861:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.227867:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575243.227887:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575243.227897:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569939456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575243.227902:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569939584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.227917:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.227919:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575243.227921:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.227924:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11996 00000100:00080000:1.0:1788575243.227926:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.227930:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575243.227933:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575243.227944:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569939712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.227950:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.227953:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575243.227954:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.227957:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11996 00000100:00080000:1.0:1788575243.227959:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575243.227961:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575243.227964:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575243.227971:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569939840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575243.227976:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575243.227978:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575243.227979:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575243.227981:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 11996 00000100:00080000:1.0:1788575243.227986:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575243.227987:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575243.227991:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575243.227995:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575243.228003:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569939968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575243.228038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff920a52890380 x1875444569939456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575243.228059:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569939456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575243.228072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575243.228077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575243.228081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569939456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575243.228108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575243.228129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575243.228141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff920a52891f80 x1875444569939584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575243.228168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569939584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575243.228179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575243.228183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575243.228187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575243.228193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575243.228213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575243.228224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff920a52891880 x1875444569939712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575243.228236:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569939712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575243.228245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575243.228248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575243.228252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575243.228257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575243.228275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575243.228284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff920a52893480 x1875444569939840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575243.228296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569939840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575243.228312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575243.228316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575243.228319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575243.228325:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575243.228343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575243.228354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575243/real 1788575243] req@ffff920a52891500 x1875444569939968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575243.228366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569939968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575243.228374:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575243.228378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575243.228381:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569939968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575243.675815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575244.251707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575244.699757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575245.275719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575245.723786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575246.299734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575246.747770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575247.323722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575247.771824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575248.347706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575248.347734:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.347748:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575248.347754:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.347768:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12001 00000100:00080000:1.0:1788575248.347782:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.347792:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575248.347844:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569940096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.347875:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.347880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575248.347882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.347887:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12001 00000100:00080000:1.0:1788575248.347892:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.347898:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575248.347909:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575248.347923:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569940096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575248.347931:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569940224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.347961:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.347965:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575248.347967:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.347973:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12001 00000100:00080000:1.0:1788575248.347979:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.347984:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575248.347997:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788575248.348010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575248.348013:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569940352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575248.348026:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575248.348031:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575248.348033:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575248.348038:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12001 00000100:00080000:1.0:1788575248.348043:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.348049:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575248.348057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff920a52890e00 x1875444569940096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575248.348062:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575248.348089:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569940480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575248.348090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569940096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.348166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575248.348167:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575248.348179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575248.348180:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575248.348187:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575248.348188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569940096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575248.348211:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12001 00000100:00080000:1.0:1788575248.348219:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575248.348224:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575248.348230:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575248.348233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575248.348253:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569940608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575248.348270:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575248.348287:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff920a52893800 x1875444569940224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.348297:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569940224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.348308:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.348311:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575248.348313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575248.348318:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575248.348332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575248.348338:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff920a52892d80 x1875444569940352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.348346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569940352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.348351:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.348353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575248.348355:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575248.348358:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575248.348369:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575248.348376:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff920a52892300 x1875444569940480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.348387:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569940480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.348395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.348398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575248.348402:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575248.348420:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575248.348438:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575248.348446:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575248/real 1788575248] req@ffff920a52891180 x1875444569940608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575248.348453:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569940608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575248.348459:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575248.348461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575248.348463:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575248.795764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575249.371699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575249.819738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575250.395670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575250.843798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575251.419704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575251.867757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575252.443707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575252.891704:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575253.467663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575253.467679:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575253.467701:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575253.467706:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575253.467718:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12006 00000100:00080000:0.0:1788575253.467729:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575253.467766:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575253.467805:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569940736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575253.467833:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575253.467837:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575253.467839:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575253.467845:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12006 00000100:00080000:0.0:1788575253.467849:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575253.467855:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575253.467865:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575253.467874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569940736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575253.467882:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569940864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575253.467895:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575253.467899:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575253.467901:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575253.467906:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12006 00000100:00080000:0.0:1788575253.467910:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575253.467915:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575253.467920:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575253.467932:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569940992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575253.467942:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575253.467944:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575253.467950:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575253.467952:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575253.467958:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12006 00000100:00080000:0.0:1788575253.467963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575253.467968:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575253.467974:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575253.467975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff920a48353480 x1875444569940736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575253.467990:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569940736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575253.467997:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575253.468000:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575253.468005:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569940736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575253.468008:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569941120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575253.468022:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575253.468027:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575253.468027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940864:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575253.468033:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575253.468040:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12006 00000100:00080000:0.0:1788575253.468046:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575253.468051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575253.468052:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575253.468061:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575253.468065:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff920a48351c00 x1875444569940864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575253.468074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569940864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575253.468080:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569941248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575253.468082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575253.468089:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575253.468095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575253.468103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575253.468119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575253.468128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff920a48351500 x1875444569940992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575253.468136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569940992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575253.468141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575253.468143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575253.468145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569940992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575253.468148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575253.468159:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575253.468165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff920a48353b80 x1875444569941120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575253.468172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569941120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575253.468177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575253.468179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575253.468181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575253.468191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575253.468200:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575253.468206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575253/real 1788575253] req@ffff920a48350700 x1875444569941248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575253.468213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569941248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575253.468218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575253.468220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575253.468221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575253.915734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575254.491726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575254.939776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575255.515659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575255.963731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575256.539692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575256.987773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575257.563730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575258.011793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575258.587694:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575258.587697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575258.587706:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575258.587717:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575258.587730:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12011 00000100:00080000:0.0:1788575258.587744:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575258.587753:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575258.587799:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569941376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575258.587824:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575258.587829:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575258.587831:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575258.587843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12011 00000100:00080000:0.0:1788575258.587848:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575258.587852:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575258.587861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569941376:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575258.587868:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575258.587880:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569941504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575258.587890:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575258.587893:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575258.587895:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575258.587899:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12011 00000100:00080000:0.0:1788575258.587903:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575258.587906:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575258.587910:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575258.587919:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569941632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575258.587927:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575258.587931:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575258.587933:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575258.587936:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12011 00000800:00000100:3.0:1788575258.587936:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575258.587939:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575258.587943:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575258.587948:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575258.587959:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569941760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575258.587967:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575258.587971:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575258.587972:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575258.587976:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12011 00000100:00080000:0.0:1788575258.587979:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575258.587979:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575258] req@ffff920a48350380 x1875444569941376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575258.587983:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575258.587988:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575258.587997:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569941888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575258.587999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569941376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575258.588012:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575258.588017:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575258.588021:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569941376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575258.588050:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575258.588076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575258.588091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575258] req@ffff920a48353800 x1875444569941504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575258.588105:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569941504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575258.588115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575258.588117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575258.588119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575258.588124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575258.588146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575258.588159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575258] req@ffff920a48350a80 x1875444569941632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575258.588172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569941632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575258.588185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575258.588188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575258.588190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575258.588194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575258.588216:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575258.588228:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575258] req@ffff920a48350700 x1875444569941760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575258.588241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569941760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575258.588249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575258.588253:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575258.588256:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575258.588263:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575258.588286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575258.588298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575258/real 1788575258] req@ffff920a48353b80 x1875444569941888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575258.588311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569941888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575258.588319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575258.588322:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575258.588327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569941888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575259.035738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575259.611737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575260.059829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575260.635718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575261.083831:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575261.659721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575262.107768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575262.683730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575263.131804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575263.707703:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575263.707717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575263.707718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575263.707726:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575263.707744:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12017 00000100:00080000:0.0:1788575263.707760:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575263.707769:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575263.707823:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569942016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575263.707862:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575263.707867:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575263.707869:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575263.707874:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12017 00000100:00080000:0.0:1788575263.707879:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575263.707884:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575263.707895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575263.707911:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569942144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575263.707923:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569942016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575263.707942:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575263.707947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575263.707949:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575263.707955:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12017 00000100:00080000:0.0:1788575263.707960:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575263.707965:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575263.707971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575263.707985:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569942272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575263.707997:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575263.708001:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575263.708003:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575263.708012:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575263.708018:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12017 00000100:00080000:0.0:1788575263.708024:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575263.708030:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575263.708037:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575263.708052:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569942400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575263.708064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575263/real 1788575263] req@ffff920a48351500 x1875444569942016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575263.708101:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575263.708107:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575263.708110:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575263.708117:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12017 00000100:00100000:3.0:1788575263.708121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569942016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575263.708124:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575263.708133:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575263.708146:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575263.708155:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575263.708164:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569942528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575263.708165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575263.708185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569942016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575263.708217:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575263.708244:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575263.708259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575263/real 1788575263] req@ffff920a48351c00 x1875444569942144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575263.708272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569942144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575263.708281:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575263.708285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575263.708289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575263.708296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575263.708336:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575263.708350:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575263/real 1788575263] req@ffff920a48353480 x1875444569942272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575263.708364:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569942272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575263.708372:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575263.708376:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575263.708380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575263.708387:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575263.708410:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575263.708422:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575263/real 1788575263] req@ffff920a48353b80 x1875444569942400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575263.708435:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569942400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575263.708444:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575263.708447:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575263.708451:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575263.708458:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575263.708481:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575263.708493:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575263/real 1788575263] req@ffff920a48350700 x1875444569942528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575263.708506:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569942528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575263.708514:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575263.708518:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575263.708522:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575264.155793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575264.731697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575265.179822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575265.755693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575266.203782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575266.779701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575267.227790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575267.803665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575268.251759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575268.827733:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575268.827736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575268.827749:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575268.827755:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575268.827770:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12022 00000100:00080000:2.0:1788575268.827784:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575268.827795:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575268.827847:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569942656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575268.827882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575268.827887:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575268.827889:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575268.827895:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12022 00000100:00080000:2.0:1788575268.827900:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575268.827906:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575268.827929:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575268.827933:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569942656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575268.827954:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569942784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575268.827971:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575268.827976:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575268.827978:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575268.827984:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12022 00000100:00080000:2.0:1788575268.827992:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575268.827998:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788575268.828003:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575268.828011:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575268.828036:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569942912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575268.828043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575268/real 1788575268] req@ffff920a60ac4e00 x1875444569942656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575268.828067:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575268.828083:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575268.828090:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575268.828101:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12022 00000100:00100000:3.0:1788575268.828101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569942656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575268.828116:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575268.828121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575268.828140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575268.828142:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575268.828146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569942656:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575268.828156:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575268.828176:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575268.828179:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569943040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575268.828196:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575268.828201:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575268.828208:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575268.828208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575268.828223:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12022 00000100:00100000:3.0:1788575268.828230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575268/real 1788575268] req@ffff920a60ac7800 x1875444569942784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575268.828232:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575268.828247:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575268.828252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569942784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575268.828257:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575268.828271:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575268.828278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575268.828284:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569943168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575268.828296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575268.828309:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575268.828334:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575268.828344:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575268/real 1788575268] req@ffff920a60ac4380 x1875444569942912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575268.828352:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569942912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575268.828358:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575268.828361:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575268.828363:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569942912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575268.828374:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575268.828386:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575268.828393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575268/real 1788575268] req@ffff920a60ac5c00 x1875444569943040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575268.828400:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569943040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575268.828406:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575268.828408:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575268.828410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575268.828414:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575268.828424:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575268.828430:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575268/real 1788575268] req@ffff920a60ac4000 x1875444569943168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575268.828446:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569943168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575268.828452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575268.828454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575268.828456:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575269.275766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575269.851699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575270.299781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575270.875679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575271.323744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575271.899730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575272.347841:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575272.923716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575273.371732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575273.947679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575273.947680:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575273.947689:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575273.947694:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575273.947703:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12027 00000100:00080000:2.0:1788575273.947711:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575273.947718:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575273.947754:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569943296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575273.947781:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575273.947786:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575273.947789:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575273.947794:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12027 00000100:00080000:2.0:1788575273.947799:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575273.947803:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575273.947808:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569943296:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575273.947815:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575273.947833:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569943424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575273.947845:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575273.947849:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575273.947851:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575273.947856:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12027 00000100:00080000:2.0:1788575273.947860:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575273.947864:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575273.947870:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575273.947883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569943552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575273.947888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943424:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575273.947905:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575273.947910:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575273.947912:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575273.947916:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12027 00000100:00080000:2.0:1788575273.947919:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575273.947923:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575273.947928:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575273.947940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569943680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575273.947950:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575273.947953:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575273.947956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575273.947970:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12027 00000100:00080000:2.0:1788575273.947974:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575273.947979:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943552:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575273.947981:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575273.947989:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575273.948002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569943808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575273.948021:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575273.948054:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575274.395763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575274.971705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575275.419720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575275.995729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575276.443749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575277.019803:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788575277.019822:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788575277.019893:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575277.019907:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575277.019948:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575277.019960:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575277.019967:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575277.019974:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788575277.020039:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575273/real 1788575277] req@ffff920a60ac6a00 x1875444569943808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575277.020076:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569943808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575277.020087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575277.020092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575277.020097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943808:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575277.020105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575273/real 1788575277] req@ffff920a60ac4700 x1875444569943552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575277.020116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569943552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575277.020123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575277.020127:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575277.020130:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575277.020140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575273/real 1788575277] req@ffff920a60ac4a80 x1875444569943680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575277.020150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569943680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575277.020157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575277.020160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575277.020164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575277.020183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575273/real 1788575277] req@ffff920a60ac6300 x1875444569943424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575277.020196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569943424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575277.020203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575277.020208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575277.020210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569943424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575277.020216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575273/real 1788575277] req@ffff920a60ac5500 x1875444569943296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575277.020228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569943296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575298 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575277.020234:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575277.020237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575277.020239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569943296:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575277.467738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575278.043718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575278.491788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575279.067686:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.067696:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575279.067700:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.067710:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12032 00000100:00080000:2.0:1788575279.067728:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575279.067733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575279.067734:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575279.067766:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569943936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.067788:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.067790:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.067791:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.067794:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12032 00000100:00080000:2.0:1788575279.067807:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.067810:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575279.067818:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.067827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569944064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.067837:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.067840:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575279.067841:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575279.067857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569943936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575279.067857:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12032 00000100:00080000:2.0:1788575279.067864:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.067870:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575279.067875:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575279.067889:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569944192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.067899:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575279.067903:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575279.067906:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.067910:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12032 00000100:00080000:2.0:1788575279.067914:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575279.067917:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575279.067922:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575279.067924:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575279.067937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569944320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575279.067947:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575279.067957:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff920a60ac7100 x1875444569943936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575279.067966:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575279.067970:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575279.067977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12032 00000100:00080000:2.0:1788575279.067981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575279.067985:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569943936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575279.067989:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575279.067998:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575279.068010:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569944448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575279.068020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.068027:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575279.068032:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569943936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575279.068040:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575279.068066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575279.068079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff920a60ac5180 x1875444569944064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.068090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569944064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.068097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.068100:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575279.068103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575279.068127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575279.068146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575279.068156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff920a60ac5500 x1875444569944192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.068166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569944192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.068173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.068176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575279.068178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575279.068183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575279.068201:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575279.068211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff920a60ac6300 x1875444569944320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.068220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569944320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.068227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.068230:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575279.068233:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575279.068238:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575279.068252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575279.068261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575279/real 1788575279] req@ffff920a60ac4a80 x1875444569944448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575279.068276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569944448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575279.068282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575279.068285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575279.068288:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575279.515743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575280.091805:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575280.539778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575281.115687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575281.563757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575282.139681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575282.587766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575283.163696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575283.611763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575284.187694:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575284.187698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575284.187706:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575284.187711:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.187720:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12037 00000100:00080000:3.0:1788575284.187729:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.187737:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575284.187770:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569944576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.187798:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.187802:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575284.187804:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.187807:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12037 00000100:00080000:3.0:1788575284.187811:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.187815:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575284.187824:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575284.187829:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569944576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575284.187839:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569944704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575284.187850:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.187853:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575284.187855:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.187859:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12037 00000100:00080000:3.0:1788575284.187863:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.187867:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575284.187872:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575284.187884:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569944832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575284.187891:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575284.187898:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575284.187903:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575284.187905:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575284.187910:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12037 00000100:00080000:3.0:1788575284.187915:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.187919:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575284.187925:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575284.187928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff920a529b2300 x1875444569944576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575284.187939:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569944960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575284.187944:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569944576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575284.187960:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575284.187962:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575284.187967:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575284.187968:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788575284.187972:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575284.187974:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569944576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575284.187980:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12037 00000100:00080000:3.0:1788575284.187984:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575284.187988:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575284.187993:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575284.187997:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575284.188007:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569945088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575284.188026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575284.188041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff920a529b0700 x1875444569944704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.188050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569944704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.188056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.188058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575284.188060:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575284.188063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575284.188076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575284.188082:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff920a529b0000 x1875444569944832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.188088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569944832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.188093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.188094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575284.188097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575284.188102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575284.188120:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575284.188129:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff920a529b1f80 x1875444569944960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.188141:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569944960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.188151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.188153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575284.188155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569944960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575284.188166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575284.188178:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575284.188185:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575284/real 1788575284] req@ffff920a529b3100 x1875444569945088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575284.188191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569945088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575284.188195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575284.188197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575284.188199:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575284.635766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575285.211715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575285.659714:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575286.235668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575286.683750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575287.259653:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575287.707717:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575288.283695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575288.731719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575289.307650:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.307658:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575289.307662:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575289.307666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575289.307671:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12042 00000100:00080000:3.0:1788575289.307680:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.307686:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575289.307720:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569945216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.307749:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.307752:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575289.307753:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.307756:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12042 00000100:00080000:3.0:1788575289.307759:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.307762:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575289.307770:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575289.307780:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569945216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575289.307782:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569945344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.307794:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.307796:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575289.307797:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.307800:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12042 00000100:00080000:3.0:1788575289.307802:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.307804:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575289.307807:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575289.307814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569945472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.307824:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.307827:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575289.307828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.307830:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12042 00000100:00080000:3.0:1788575289.307838:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.307841:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575289.307844:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788575289.307845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575289.307855:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569945600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575289.307861:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575289.307863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575289.307865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575289.307867:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12042 00000100:00080000:3.0:1788575289.307870:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575289.307872:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575289.307875:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575289.307877:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff920a529b1c00 x1875444569945216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575289.307884:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569945728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575289.307911:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569945216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575289.307924:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575289.307927:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575289.307930:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569945216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575289.307958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575289.307982:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575289.307995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff920a529b2a00 x1875444569945344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575289.308008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569945344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575289.308016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575289.308019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575289.308022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575289.308028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575289.308049:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575289.308060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff920a529b3480 x1875444569945472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575289.308071:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569945472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575289.308079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575289.308082:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575289.308086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575289.308092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575289.308112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575289.308123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff920a529b2d80 x1875444569945600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575289.308135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569945600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575289.308142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575289.308146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575289.308149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575289.308155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575289.308175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575289.308187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575289/real 1788575289] req@ffff920a529b3100 x1875444569945728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575289.308198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569945728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575289.308205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575289.308208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575289.308211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575289.755746:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575290.331667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575290.779731:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575291.355681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575291.803756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575292.379723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575292.827729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575293.403684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575293.851734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575294.427654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575294.427664:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.427669:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575294.427673:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.427681:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12047 00000100:00080000:3.0:1788575294.427690:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.427696:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575294.427727:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569945856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575294.427745:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.427747:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575294.427748:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.427752:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12047 00000100:00080000:3.0:1788575294.427754:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.427757:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575294.427765:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575294.427771:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569945856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575294.427776:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569945984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575294.427784:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.427787:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575294.427789:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.427794:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12047 00000100:00080000:3.0:1788575294.427797:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.427801:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575294.427805:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575294.427817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569946112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575294.427827:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575294.427830:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575294.427835:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575294.427837:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.427841:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12047 00000100:00080000:3.0:1788575294.427845:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.427848:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575294.427852:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575294.427861:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569946240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575294.427870:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff920a529b1f80 x1875444569945856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575294.427874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575294.427883:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575294.427886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575294.427889:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12047 00000100:00080000:3.0:1788575294.427892:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575294.427896:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575294.427898:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569945856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575294.427900:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575294.427911:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575294.427912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569946368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575294.427917:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575294.427923:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569945856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575294.427953:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575294.427976:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575294.427996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff920a529b0000 x1875444569945984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.428009:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569945984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.428016:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.428019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575294.428022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569945984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575294.428028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575294.428047:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575294.428057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff920a529b0700 x1875444569946112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.428068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569946112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.428075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.428078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575294.428081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575294.428086:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575294.428103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575294.428112:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff920a529b2300 x1875444569946240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.428122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569946240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.428129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.428132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575294.428135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575294.428141:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575294.428156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575294.428177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575294/real 1788575294] req@ffff920a529b0380 x1875444569946368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575294.428189:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569946368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575294.428206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575294.428208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575294.428211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575294.875703:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575295.451716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575295.899741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575296.475690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575296.923799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575297.499687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575297.947740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575298.523706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575298.971797:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575299.547711:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575299.547723:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575299.547728:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575299.547740:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12052 00000100:00080000:3.0:1788575299.547752:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575299.547757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575299.547762:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575299.547810:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569946496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575299.547835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575299.547838:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575299.547840:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575299.547843:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12052 00000100:00080000:3.0:1788575299.547846:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575299.547848:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575299.547858:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575299.547870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569946624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575299.547878:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575299.547884:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575299.547885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569946496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575299.547886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575299.547893:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12052 00000100:00080000:3.0:1788575299.547896:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575299.547899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575299.547903:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575299.547912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569946752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575299.547924:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575299.547927:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575299.547928:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575299.547930:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12052 00000100:00080000:3.0:1788575299.547934:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575299.547936:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575299.547939:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575299.547946:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569946880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575299.547952:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575299.547955:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575299.547956:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575299.547958:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12052 00000100:00080000:3.0:1788575299.547961:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575299.547962:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575299.547967:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575299.547971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575299.547979:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569947008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575299.548005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff920a529b0e00 x1875444569946496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575299.548021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569946496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575299.548030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575299.548033:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575299.548036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569946496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575299.548058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575299.548076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575299.548089:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff920a529b1500 x1875444569946624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575299.548102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569946624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575299.548112:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575299.548116:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575299.548120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575299.548127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575299.548163:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575299.548178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff920a529b1180 x1875444569946752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575299.548191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569946752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575299.548200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575299.548203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575299.548205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575299.548210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575299.548227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575299.548234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff920a529b3b80 x1875444569946880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575299.548242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569946880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575299.548247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575299.548248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575299.548251:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569946880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575299.548255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575299.548265:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575299.548271:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575299/real 1788575299] req@ffff920a529b0380 x1875444569947008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575299.548277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569947008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575299.548282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575299.548284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575299.548286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575299.995793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575300.571709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575301.019795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575301.595684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575302.043732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575302.619686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575303.067766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575303.643688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575304.091771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575304.667707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575304.667718:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575304.667724:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575304.667728:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575304.667753:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12058 00000100:00080000:3.0:1788575304.667763:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575304.667770:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575304.667801:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569947136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575304.667820:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575304.667823:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575304.667825:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575304.667828:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12058 00000100:00080000:3.0:1788575304.667830:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575304.667833:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575304.667841:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575304.667851:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569947264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575304.667857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569947136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575304.667860:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575304.667868:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575304.667870:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575304.667873:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12058 00000100:00080000:3.0:1788575304.667876:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575304.667878:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575304.667881:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575304.667890:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569947392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575304.667897:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575304.667900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575304.667901:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575304.667904:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12058 00000100:00080000:3.0:1788575304.667914:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575304.667917:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575304.667921:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575304.667929:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569947520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575304.667937:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575304.667941:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575304.667945:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575304.667946:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575304.667950:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12058 00000100:00080000:3.0:1788575304.667953:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575304.667956:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575304.667959:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575304.667966:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569947648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575304.667983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575304/real 1788575304] req@ffff920a529b2300 x1875444569947136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575304.668020:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569947136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575304.668033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575304.668037:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575304.668042:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569947136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575304.668067:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575304.668092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575304.668103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575304/real 1788575304] req@ffff920a529b0700 x1875444569947264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575304.668112:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569947264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575304.668118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575304.668120:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575304.668123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575304.668127:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575304.668140:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575304.668145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575304/real 1788575304] req@ffff920a529b0000 x1875444569947392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575304.668153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569947392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575304.668159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575304.668160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575304.668162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575304.668166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575304.668176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575304.668183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575304/real 1788575304] req@ffff920a529b1f80 x1875444569947520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575304.668190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569947520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575304.668195:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575304.668196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575304.668198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575304.668201:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575304.668211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575304.668216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575304/real 1788575304] req@ffff920a529b3100 x1875444569947648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575304.668223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569947648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575304.668227:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575304.668229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575304.668231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575305.115713:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575305.691685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575306.139795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575306.715720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575307.163714:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575307.739705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575308.187742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575308.763674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575309.211717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575309.787700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575309.787719:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575309.787725:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575309.787729:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575309.787738:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12063 00000100:00080000:0.0:1788575309.787748:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575309.787754:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575309.787786:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569947776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575309.787810:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575309.787813:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575309.787814:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575309.787819:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12063 00000100:00080000:0.0:1788575309.787834:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575309.787838:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575309.787848:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575309.787858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569947776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575309.787870:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569947904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575309.787880:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575309.787883:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575309.787884:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575309.787887:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12063 00000100:00080000:0.0:1788575309.787891:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575309.787894:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575309.787897:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575309.787908:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569948032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575309.787919:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575309.787924:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575309.787926:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575309.787933:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12063 00000800:00000100:3.0:1788575309.787937:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575309.787945:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575309.787953:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575309.787961:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575309.787977:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569948160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575309.787989:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575309.787990:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575309/real 1788575309] req@ffff920a48350a80 x1875444569947776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575309.787998:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575309.788002:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575309.788012:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12063 00000100:00080000:0.0:1788575309.788018:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575309.788024:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575309.788027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569947776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575309.788032:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575309.788054:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569948288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575309.788120:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575309.788125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575309.788129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569947776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575309.788153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575309.788180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575309.788194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575309/real 1788575309] req@ffff920a48353800 x1875444569947904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575309.788204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569947904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575309.788210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575309.788212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575309.788215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569947904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575309.788345:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575309.788366:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575309.788377:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575309/real 1788575309] req@ffff920a48350380 x1875444569948032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575309.788427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569948032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575309.788434:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575309.788437:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575309.788440:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575309.788445:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575309.788461:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575309.788469:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575309/real 1788575309] req@ffff920a48350700 x1875444569948160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575309.788477:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569948160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575309.788482:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575309.788484:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575309.788486:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575309.788489:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575309.788500:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575309.788506:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575309/real 1788575309] req@ffff920a48353b80 x1875444569948288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575309.788513:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569948288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575309.788518:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575309.788519:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575309.788521:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575310.235763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575310.811731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575311.259820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575311.835683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575312.283720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575312.859693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575313.307805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575313.883661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575314.331728:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575314.907714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575314.907715:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575314.907725:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575314.907730:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575314.907741:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12068 00000100:00080000:3.0:1788575314.907752:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575314.907760:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575314.907796:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569948416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575314.907823:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575314.907828:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575314.907830:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575314.907834:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12068 00000100:00080000:3.0:1788575314.907837:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575314.907841:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575314.907850:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575314.907864:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569948544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575314.907876:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575314.907880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575314.907883:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575314.907886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569948416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575314.907890:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12068 00000100:00080000:3.0:1788575314.907895:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575314.907901:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575314.907906:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575314.907916:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569948672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575314.907933:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575314.907936:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575314.907938:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575314.907942:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12068 00000100:00080000:3.0:1788575314.907945:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575314.907948:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575314.907953:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575314.907961:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569948800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575314.907967:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575314.907974:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575314.907977:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575314.907978:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575314.907981:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12068 00000100:00080000:3.0:1788575314.907984:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575314.907987:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575314.907990:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575314.907999:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569948928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575314.908007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575314/real 1788575314] req@ffff920a529b2d80 x1875444569948416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575314.908024:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569948416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575314.908033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575314.908036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575314.908039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569948416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575314.908058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575314.908076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575314.908084:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575314/real 1788575314] req@ffff920a529b3480 x1875444569948544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575314.908094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569948544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575314.908103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575314.908107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575314.908110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575314.908120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575314.908143:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575314.908168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575314/real 1788575314] req@ffff920a529b2a00 x1875444569948672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575314.908178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569948672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575314.908184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575314.908187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575314.908189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575314.908194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575314.908215:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575314.908223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575314/real 1788575314] req@ffff920a529b1c00 x1875444569948800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575314.908231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569948800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575314.908236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575314.908239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575314.908241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575314.908244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575314.908256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575314.908261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575314/real 1788575314] req@ffff920a529b3100 x1875444569948928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575314.908273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569948928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575314.908281:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575314.908283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575314.908287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569948928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575315.355784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575315.931686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575316.379785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575316.955712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575317.403771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575317.979681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575318.427740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575319.003713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575319.451782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575320.027667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575320.027667:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.027677:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575320.027681:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.027689:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12073 00000100:00080000:3.0:1788575320.027698:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.027706:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575320.027740:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569949056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.027763:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.027767:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575320.027772:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.027776:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12073 00000100:00080000:3.0:1788575320.027778:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.027781:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575320.027790:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575320.027799:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569949184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.027807:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.027809:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575320.027810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.027813:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12073 00000100:00080000:3.0:1788575320.027815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.027817:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575320.027820:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575320.027823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569949056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575320.027829:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569949312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.027837:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.027840:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575320.027841:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575320.027844:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12073 00000100:00080000:3.0:1788575320.027846:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.027848:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575320.027858:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575320.027866:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569949440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575320.027873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575320.027876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575320.027878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575320.027881:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575320.027886:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12073 00000100:00080000:3.0:1788575320.027891:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575320.027896:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575320.027901:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575320.027913:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff920a529b1f80 x1875444569949056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575320.027918:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569949568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575320.027935:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569949056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575320.027944:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575320.027948:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575320.027951:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569949056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575320.027956:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575320.027974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575320.027987:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff920a529b0000 x1875444569949184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575320.028002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569949184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575320.028011:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575320.028015:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575320.028019:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575320.028044:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575320.028067:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575320.028079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff920a529b0700 x1875444569949312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575320.028093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569949312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575320.028102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575320.028106:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575320.028109:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575320.028116:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575320.028136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575320.028147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff920a529b2300 x1875444569949440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575320.028158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569949440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575320.028174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575320.028177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575320.028181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575320.028187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575320.028209:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575320.028221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575320/real 1788575320] req@ffff920a529b0380 x1875444569949568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575320.028235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569949568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575320.028244:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575320.028247:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575320.028251:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575320.475752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575321.051669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575321.499763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575322.075685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575322.523716:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575323.099684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575323.547803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575324.123725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575324.571719:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575325.147699:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575325.147710:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1788575325.147710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575325.147716:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575325.147727:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12078 00000100:00080000:2.0:1788575325.147737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575325.147746:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575325.147784:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569949696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575325.147816:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575325.147820:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575325.147822:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575325.147827:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12078 00000100:00080000:2.0:1788575325.147832:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575325.147838:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575325.147849:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575325.147852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569949696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575325.147866:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569949824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575325.147875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575325.147878:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575325.147879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575325.147889:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12078 00000100:00080000:2.0:1788575325.147893:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575325.147897:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575325.147901:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575325.147911:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569949952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575325.147919:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575325.147921:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575325.147925:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575325.147926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575325.147931:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12078 00000100:00080000:2.0:1788575325.147936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575325.147940:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575325.147946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575325.147959:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569950080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575325.147962:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575325] req@ffff920a60ac4700 x1875444569949696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575325.147985:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575325.147991:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575325.147994:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575325.147995:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569949696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575325.148003:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12078 00000100:00080000:2.0:1788575325.148011:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575325.148018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575325.148019:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788575325.148028:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:2.0:1788575325.148029:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575325.148036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569949696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575325.148046:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569950208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575325.148072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575325.148099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575325.148113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575325] req@ffff920a60ac6a00 x1875444569949824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575325.148128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569949824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575325.148137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575325.148141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575325.148144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575325.148151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575325.148176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575325.148188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575325] req@ffff920a60ac4000 x1875444569949952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575325.148201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569949952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575325.148210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575325.148214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575325.148218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569949952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575325.148225:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575325.148246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575325.148259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575325] req@ffff920a60ac5c00 x1875444569950080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575325.148272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569950080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575325.148280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575325.148284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575325.148287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575325.148293:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575325.148313:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575325.148325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575325/real 1788575325] req@ffff920a60ac4380 x1875444569950208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575325.148534:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569950208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575325.148543:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575325.148547:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575325.148550:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575325.595762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575326.171700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575326.619725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575327.195681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575327.643738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575328.219677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575328.667782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575329.243715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575329.691760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575330.267727:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575330.267739:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1788575330.267740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575330.267744:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575330.267757:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12083 00000100:00080000:0.0:1788575330.267769:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575330.267778:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575330.267814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569950336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575330.267837:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575330.267841:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575330.267843:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575330.267848:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12083 00000100:00080000:0.0:1788575330.267853:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575330.267857:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575330.267865:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575330.267875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569950336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575330.267880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569950464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575330.267890:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575330.267893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575330.267894:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575330.267897:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12083 00000100:00080000:0.0:1788575330.267900:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575330.267902:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575330.267906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575330.267914:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569950592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575330.267921:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575330.267924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575330.267925:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575330.267928:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12083 00000100:00080000:0.0:1788575330.267931:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575330.267936:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575330.267936:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575330.267943:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575330.267953:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569950720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575330.267959:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575330.267962:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575330.267963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575330.267967:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12083 00000100:00080000:0.0:1788575330.267970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575330.267974:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff920a48353480 x1875444569950336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575330.267975:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575330.267982:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575330.267991:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569950848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575330.267999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569950336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.268009:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.268012:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575330.268017:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569950336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575330.268034:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575330.268050:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575330.268058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff920a48351c00 x1875444569950464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.268066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569950464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.268071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.268073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575330.268075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575330.268078:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575330.268096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575330.268108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff920a48351500 x1875444569950592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.268121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569950592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.268130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.268134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575330.268137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575330.268144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575330.268165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575330.268203:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff920a48353b80 x1875444569950720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.268212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569950720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.268217:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.268220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575330.268222:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575330.268226:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575330.268240:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575330.268246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575330/real 1788575330] req@ffff920a48350700 x1875444569950848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575330.268253:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569950848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575330.268263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575330.268265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575330.268267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569950848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575330.715806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575331.291685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575331.739733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575332.315739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575332.763834:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575333.339685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575333.787774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575334.363726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575334.811771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575335.387662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575335.387687:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575335.387696:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575335.387702:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575335.387717:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12088 00000100:00080000:1.0:1788575335.387730:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575335.387740:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575335.387805:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569950976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575335.387841:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575335.387845:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575335.387847:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575335.387853:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12088 00000100:00080000:1.0:1788575335.387858:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575335.387863:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575335.387876:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575335.387889:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569950976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575335.387895:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569951104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575335.387912:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575335.387916:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575335.387919:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575335.387924:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12088 00000100:00080000:1.0:1788575335.387929:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575335.387934:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575335.387939:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575335.387953:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569951232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575335.387965:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575335.387969:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575335.387971:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575335.387975:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12088 00000100:00080000:1.0:1788575335.387979:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575335.387984:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575335.387991:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575335.388003:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569951360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575335.388014:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575335.388018:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575335.388020:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575335.388025:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12088 00000100:00080000:1.0:1788575335.388030:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575335.388034:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575335.388040:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575335.388054:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569951488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575335.388080:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575335.388148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff920a52890700 x1875444569950976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575335.388179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569950976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575335.388191:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575335.388196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575335.388200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569950976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575335.388227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575335.388252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575335.388262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff920a52893100 x1875444569951104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575335.388271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569951104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575335.388277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575335.388279:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575335.388282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575335.388285:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575335.388300:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575335.388306:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff920a52891180 x1875444569951232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575335.388314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569951232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575335.388321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575335.388325:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575335.388328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575335.388334:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575335.388356:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575335.388368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff920a52892300 x1875444569951360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575335.388381:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569951360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575335.388389:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575335.388393:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575335.388396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575335.388402:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575335.388424:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575335.388436:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575335/real 1788575335] req@ffff920a52892d80 x1875444569951488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575335.388454:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569951488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575335.388460:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575335.388462:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575335.388465:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575335.835829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575336.411737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575336.859807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575337.435716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575337.883762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575338.459724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575338.907761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575339.483699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575339.931779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575340.507724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575340.507758:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575340.507770:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575340.507775:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575340.507787:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12093 00000100:00080000:0.0:1788575340.507797:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575340.507805:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575340.507856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569951616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575340.507885:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575340.507890:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575340.507892:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575340.507897:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12093 00000100:00080000:0.0:1788575340.507902:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575340.507908:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575340.507919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575340.507927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569951616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575340.507939:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569951744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575340.507965:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575340.507969:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575340.507971:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575340.507978:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12093 00000100:00080000:0.0:1788575340.507983:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575340.507987:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575340.507993:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575340.508021:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569951872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575340.508033:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575340.508036:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575340.508038:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575340.508042:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12093 00000100:00080000:0.0:1788575340.508047:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575340.508053:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575340.508064:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575340.508083:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575340.508085:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569952000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575340.508119:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575340.508125:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575340.508128:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575340.508133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951872:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575340.508136:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12093 00000100:00080000:0.0:1788575340.508146:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575340.508154:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575340.508172:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575340.508177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575340.508195:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569952128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575340.508218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575340.955815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575341.531699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575341.979757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575342.555716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575343.003760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575343.579721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575343.579883:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575343.579926:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575343.579942:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575343.580020:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575343.580031:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575343.580040:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575343.580049:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788575343.580120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575343] req@ffff920a48353b80 x1875444569952128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575343.580145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569952128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575343.580159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575343.580165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575343.580170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952128:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575343.580179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575343] req@ffff920a48350700 x1875444569952000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575343.580194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569952000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575343.580204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575343.580208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575343.580212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575343.580219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575343] req@ffff920a48353800 x1875444569951744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575343.580231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569951744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575343.580239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575343.580244:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575343.580247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575343.580255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575343] req@ffff920a48350a80 x1875444569951872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575343.580267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569951872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575343.580291:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575343.580296:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575343.580299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569951872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575343.580307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575340/real 1788575343] req@ffff920a48350380 x1875444569951616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575343.580322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569951616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575343.580332:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575343.580336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575343.580339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569951616:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575344.027764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575344.603687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575345.051758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575345.627698:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575345.627704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575345.627707:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575345.627713:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575345.627722:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12099 00000100:00080000:0.0:1788575345.627730:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575345.627737:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575345.627767:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569952256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575345.627789:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575345.627793:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575345.627794:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575345.627798:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12099 00000100:00080000:0.0:1788575345.627801:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575345.627805:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575345.627815:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575345.627831:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569952384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575345.627837:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569952256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575345.627846:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575345.627851:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575345.627853:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575345.627858:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12099 00000100:00080000:0.0:1788575345.627863:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575345.627867:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575345.627873:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575345.627891:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569952512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575345.627903:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575345.627907:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575345.627910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575345.627913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575345.627918:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12099 00000100:00080000:0.0:1788575345.627926:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575345.627945:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575345.627953:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575345.627970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575345/real 1788575345] req@ffff920a48351500 x1875444569952256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575345.627973:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569952640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575345.628013:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575345.628016:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569952256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575345.628017:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575345.628022:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575345.628031:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12099 00000100:00080000:3.0:1788575345.628038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575345.628040:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575345.628047:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575345.628050:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575345.628057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569952256:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575345.628058:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575345.628079:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569952768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575345.628093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575345.628125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575345.628143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575345/real 1788575345] req@ffff920a48351c00 x1875444569952384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575345.628161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569952384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575345.628173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575345.628177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575345.628182:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575345.628191:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575345.628214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575345.628226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575345/real 1788575345] req@ffff920a48353480 x1875444569952512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575345.628241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569952512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575345.628252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575345.628256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575345.628259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575345.628267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575345.628292:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575345.628305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575345/real 1788575345] req@ffff920a48350380 x1875444569952640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575345.628319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569952640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575345.628326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575345.628329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575345.628333:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575345.628349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575345.628372:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575345.628384:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575345/real 1788575345] req@ffff920a48350a80 x1875444569952768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575345.628395:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569952768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575345.628404:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575345.628407:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575345.628411:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569952768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575346.075744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575346.651722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575347.099797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575347.675679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575348.123790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575348.699676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575349.147764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575349.723703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575350.171789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575350.747693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575350.747694:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575350.747702:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575350.747706:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575350.747714:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12104 00000100:00080000:0.0:1788575350.747723:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575350.747729:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575350.747758:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569952896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575350.747780:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575350.747783:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575350.747784:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575350.747787:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12104 00000100:00080000:0.0:1788575350.747789:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575350.747792:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575350.747801:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575350.747810:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569953024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575350.747819:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575350.747823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569952896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575350.747824:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575350.747826:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575350.747833:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12104 00000100:00080000:0.0:1788575350.747835:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575350.747838:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575350.747841:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575350.747863:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569953152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575350.747871:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575350.747874:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575350.747875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575350.747879:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12104 00000100:00080000:0.0:1788575350.747883:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575350.747886:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575350.747889:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575350.747898:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569953280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575350.747905:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575350.747910:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788575350.747912:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575350.747913:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575350.747920:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12104 00000100:00080000:0.0:1788575350.747922:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575350.747925:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575350.747928:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575350.747937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569953408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575350.747959:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575350/real 1788575350] req@ffff920a48353800 x1875444569952896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575350.747980:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569952896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575350.747991:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575350.747996:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575350.748000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569952896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575350.748032:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575350.748052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575350.748062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575350/real 1788575350] req@ffff920a48350700 x1875444569953024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575350.748079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569953024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575350.748085:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575350.748088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575350.748090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575350.748094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575350.748106:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575350.748113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575350/real 1788575350] req@ffff920a48353b80 x1875444569953152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575350.748121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569953152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575350.748130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575350.748132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575350.748134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575350.748138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575350.748149:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575350.748154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575350/real 1788575350] req@ffff920a48350a80 x1875444569953280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575350.748161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569953280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575350.748166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575350.748168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575350.748169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575350.748173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575350.748182:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575350.748187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575350/real 1788575350] req@ffff920a48350380 x1875444569953408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575350.748193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569953408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575350.748198:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575350.748200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575350.748202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575351.195758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575351.771734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575352.219766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575352.795732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575353.243799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575353.819728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575354.267788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575354.843729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575355.291760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575355.867714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575355.867764:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575355.867787:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575355.867792:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575355.867804:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12109 00000100:00080000:0.0:1788575355.867816:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575355.867827:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575355.867878:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569953536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575355.867903:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575355.867906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575355.867908:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575355.867911:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12109 00000100:00080000:0.0:1788575355.867914:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575355.867919:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575355.867933:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575355.867944:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569953536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575355.867952:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569953664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575355.867977:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575355.867982:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575355.867984:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575355.867990:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12109 00000100:00080000:0.0:1788575355.867997:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575355.868002:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788575355.868010:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575355.868014:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575355.868032:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569953792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575355.868042:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575355.868045:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575355.868047:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575355.868052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575355/real 1788575355] req@ffff920a48353480 x1875444569953536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575355.868057:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12109 00000100:00080000:0.0:1788575355.868066:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575355.868080:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575355.868085:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569953536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575355.868093:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575355.868097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575355.868102:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575355.868106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569953536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575355.868117:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569953920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575355.868133:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575355.868134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575355.868154:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575355.868158:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575355.868163:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12109 00000100:00080000:0.0:1788575355.868177:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575355.868177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575355.868191:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575355.868198:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575355.868199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575355/real 1788575355] req@ffff920a48351c00 x1875444569953664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575355.868218:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569954048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575355.868219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569953664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575355.868282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575355.868287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575355.868291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575355.868296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575355.868311:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575355.868323:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575355/real 1788575355] req@ffff920a48351500 x1875444569953792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575355.868334:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569953792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575355.868340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575355.868345:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575355.868350:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575355.868380:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575355.868413:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575355.868426:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575355/real 1788575355] req@ffff920a48350380 x1875444569953920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575355.868435:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569953920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575355.868441:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575355.868444:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575355.868446:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569953920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575355.868450:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575355.868463:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575355.868471:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575355/real 1788575355] req@ffff920a48350a80 x1875444569954048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575355.868478:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569954048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575355.868483:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575355.868484:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575355.868486:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575356.315758:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575356.891691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575357.339723:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575357.915666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575358.363790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575358.939678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575359.387753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575359.963695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575360.411825:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575360.987680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575360.987717:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575360.987724:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575360.987729:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575360.987741:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12114 00000100:00080000:1.0:1788575360.987751:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575360.987760:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575360.987796:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569954176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575360.987823:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575360.987828:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575360.987830:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575360.987836:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12114 00000100:00080000:1.0:1788575360.987840:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575360.987846:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575360.987855:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569954176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788575360.987856:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575360.987875:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569954304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575360.987892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575360.987896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575360.987898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575360.987902:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12114 00000100:00080000:1.0:1788575360.987907:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575360.987911:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575360.987916:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788575360.987930:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575360.987931:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569954432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575360.987943:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575360.987947:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575360.987950:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575360.987954:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12114 00000100:00080000:1.0:1788575360.987959:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575360.987963:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575360.987969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575360/real 1788575360] req@ffff920a52893800 x1875444569954176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575360.987970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575360.987986:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569954560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575360.987995:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569954176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575360.988001:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575360.988011:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575360.988015:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575360.988018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575360.988022:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12114 00000100:00080000:3.0:1788575360.988027:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575360.988029:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575360.988034:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569954176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575360.988039:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575360.988047:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575360.988060:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569954688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575360.988064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575360.988091:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575360.988107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575360/real 1788575360] req@ffff920a52890e00 x1875444569954304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575360.988121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569954304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575360.988130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575360.988133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575360.988137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575360.988143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575360.988158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575360.988167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575360/real 1788575360] req@ffff920a52891500 x1875444569954432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575360.988179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569954432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575360.988188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575360.988192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575360.988203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575360.988210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575360.988228:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575360.988238:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575360/real 1788575360] req@ffff920a52893480 x1875444569954560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575360.988250:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569954560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575360.988259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575360.988262:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575360.988265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575360.988279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575360.988293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575360.988302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575360/real 1788575360] req@ffff920a52891880 x1875444569954688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575360.988314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569954688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575385 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575360.988322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575360.988326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575360.988329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575361.435758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575362.011703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575362.459826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575363.035646:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575363.483725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575364.059712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575364.507756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575365.083686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575365.531732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575366.107728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575366.107736:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575366.107745:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575366.107749:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575366.107758:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12119 00000100:00080000:2.0:1788575366.107768:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575366.107774:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575366.107807:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569954816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575366.107830:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575366.107833:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575366.107835:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575366.107838:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12119 00000100:00080000:2.0:1788575366.107840:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575366.107843:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575366.107850:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575366.107860:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569954944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575366.107870:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575366.107873:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575366.107874:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575366.107877:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12119 00000100:00080000:2.0:1788575366.107881:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575366.107886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569954816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575366.107887:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575366.107893:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575366.107903:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569955072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575366.107911:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575366.107914:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575366.107915:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575366.107917:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12119 00000100:00080000:2.0:1788575366.107919:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575366.107921:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575366.107925:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575366.107933:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569955200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575366.107939:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575366.107942:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575366.107944:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575366.107947:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12119 00000100:00080000:2.0:1788575366.107950:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575366.107952:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575366.107954:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575366.107962:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569955328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575366.107993:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575366.108040:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff920a60ac7800 x1875444569954816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.108062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569954816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575366.108075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575366.108080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575366.108085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569954816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575366.108114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575366.108138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575366.108151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff920a60ac4e00 x1875444569954944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.108164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569954944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575366.108173:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575366.108176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575366.108180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569954944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575366.108187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575366.108206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575366.108217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff920a60ac4380 x1875444569955072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.108230:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569955072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575366.108238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575366.108243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575366.108246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575366.108253:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575366.108272:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575366.108283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff920a60ac5c00 x1875444569955200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.108295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569955200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575366.108303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575366.108307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575366.108310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575366.108317:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575366.108334:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575366.108354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575366/real 1788575366] req@ffff920a60ac4000 x1875444569955328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575366.108366:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569955328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575366.108375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575366.108379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575366.108382:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575366.555810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575367.131690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575367.579826:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575368.155713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575368.603774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575369.179727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575369.627784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575370.203675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575370.651705:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575371.227689:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575371.227692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575371.227701:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575371.227706:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575371.227717:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12124 00000100:00080000:3.0:1788575371.227728:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575371.227736:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575371.227774:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569955456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575371.227803:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575371.227807:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575371.227809:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575371.227815:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12124 00000100:00080000:3.0:1788575371.227819:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575371.227825:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575371.227835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569955456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575371.227837:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575371.227858:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569955584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575371.227870:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575371.227875:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575371.227878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575371.227883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12124 00000100:00080000:3.0:1788575371.227889:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575371.227893:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575371.227897:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788575371.227902:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575371.227920:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569955712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575371.227930:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff920a529b3b80 x1875444569955456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575371.227933:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575371.227945:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575371.227950:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575371.227957:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12124 00000100:00100000:1.0:1788575371.227958:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569955456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575371.227964:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575371.227970:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575371.227974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788575371.227977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575371.227983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569955456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575371.227984:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575371.228002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569955840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575371.228003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955584:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575371.228022:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575371.228026:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1788575371.228028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575371.228030:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575371.228036:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12124 00000100:00080000:3.0:1788575371.228041:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575371.228043:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff920a529b1180 x1875444569955584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575371.228049:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575371.228058:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575371.228062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569955584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.228071:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.228076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575371.228077:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569955968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575371.228080:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575371.228091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575371.228108:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575371.228117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff920a529b1500 x1875444569955712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.228130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569955712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.228139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.228143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575371.228147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575371.228163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575371.228185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575371.228196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff920a529b0e00 x1875444569955840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.228209:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569955840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.228215:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.228217:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575371.228219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575371.228223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575371.228242:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575371.228253:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575371/real 1788575371] req@ffff920a529b3b80 x1875444569955968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575371.228265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569955968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575371.228274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575371.228278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575371.228281:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569955968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575371.675791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575372.251676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575372.699767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575373.275733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575373.723733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575374.299726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575374.747761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575375.323716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575375.771750:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575376.347697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575376.347708:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.347715:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575376.347720:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.347732:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12129 00000100:00080000:3.0:1788575376.347743:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.347751:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575376.347784:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569956096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.347804:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.347807:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575376.347808:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.347812:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12129 00000100:00080000:3.0:1788575376.347815:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.347818:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575376.347826:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575376.347833:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569956096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575376.347838:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569956224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.347847:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.347850:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575376.347852:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.347854:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12129 00000100:00080000:3.0:1788575376.347857:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.347859:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575376.347862:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575376.347871:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569956352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.347881:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.347883:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575376.347885:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.347887:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12129 00000100:00080000:3.0:1788575376.347890:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575376.347892:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575376.347896:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575376.347900:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575376.347908:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569956480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575376.347914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575376.347917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575376.347918:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575376.347923:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12129 00000100:00100000:1.0:1788575376.347927:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff920a529b0380 x1875444569956096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575376.347929:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575376.347940:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575376.347947:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575376.347948:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569956096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.347961:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.347965:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575376.347967:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569956608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575376.347970:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569956096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575376.347995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575376.348013:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575376.348021:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff920a529b2300 x1875444569956224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.348029:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569956224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.348034:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.348036:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575376.348038:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575376.348042:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575376.348052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575376.348058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff920a529b0700 x1875444569956352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.348070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569956352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.348075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.348077:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575376.348079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575376.348082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575376.348094:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575376.348099:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff920a529b0000 x1875444569956480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.348106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569956480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.348111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.348113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575376.348115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575376.348118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575376.348127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575376.348133:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575376/real 1788575376] req@ffff920a529b1f80 x1875444569956608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575376.348145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569956608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575376.348159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575376.348163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575376.348166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575376.795761:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575377.371744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575377.819764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575378.395706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575378.843709:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575379.419710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575379.867780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575380.443712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575380.891727:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575381.467704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575381.467751:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575381.467759:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575381.467764:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575381.467773:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12134 00000100:00080000:0.0:1788575381.467783:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575381.467789:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575381.467820:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569956736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575381.467842:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575381.467845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575381.467847:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575381.467850:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12134 00000100:00080000:0.0:1788575381.467854:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575381.467857:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575381.467865:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575381.467878:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569956736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575381.467880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569956864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575381.467893:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575381.467896:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575381.467898:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575381.467902:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12134 00000100:00080000:0.0:1788575381.467905:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575381.467916:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575381.467920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575381.467938:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569956992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575381.467939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575381.467953:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575381.467957:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575381.467959:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575381.467965:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12134 00000100:00080000:0.0:1788575381.467970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575381.467970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff920a48353b80 x1875444569956736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575381.467979:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575381.467990:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575381.468004:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569957120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575381.468013:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575381.468017:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575381.468019:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575381.468023:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12134 00000100:00080000:0.0:1788575381.468026:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575381.468029:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575381.468033:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575381.468042:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569957248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575381.468078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569956736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575381.468090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575381.468094:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575381.468099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569956736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575381.468123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575381.468143:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575381.468153:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff920a48350700 x1875444569956864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575381.468161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569956864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575381.468167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575381.468169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575381.468171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575381.468175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575381.468186:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575381.468192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff920a48353800 x1875444569956992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575381.468199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569956992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575381.468322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575381.468328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575381.468332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569956992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575381.468384:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575381.468419:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575381.468435:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff920a48350a80 x1875444569957120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575381.468444:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569957120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575381.468450:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575381.468452:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575381.468454:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575381.468458:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575381.468470:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575381.468476:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575381/real 1788575381] req@ffff920a48350380 x1875444569957248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575381.468484:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569957248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575381.468489:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575381.468490:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575381.468493:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575381.915747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575382.491688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575382.939742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575383.515680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575383.963742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575384.539745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575384.987762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575385.563745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575386.011779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575386.587666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575386.587684:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575386.587690:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575386.587694:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575386.587707:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12139 00000100:00080000:0.0:1788575386.587720:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575386.587730:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575386.587773:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569957376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575386.587797:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575386.587801:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575386.587802:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575386.587805:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12139 00000100:00080000:0.0:1788575386.587808:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575386.587810:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575386.587820:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575386.587831:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569957504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575386.587838:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575386.587843:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575386.587844:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569957376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575386.587845:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575386.587853:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12139 00000100:00080000:0.0:1788575386.587857:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575386.587864:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575386.587868:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575386.587877:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569957632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575386.587884:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575386.587887:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575386.587889:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575386.587894:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12139 00000100:00080000:0.0:1788575386.587898:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575386.587902:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575386.587908:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575386.587926:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569957760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575386.587928:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575386.587941:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575386.587945:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575386.587947:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575386.587952:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12139 00000100:00080000:0.0:1788575386.587955:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575386.587959:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575386.587963:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575386.587973:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569957888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575386.587978:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff920a48351500 x1875444569957376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575386.588015:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569957376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575386.588026:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575386.588030:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575386.588033:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569957376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575386.588057:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575386.588076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575386.588085:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff920a48351c00 x1875444569957504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575386.588094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569957504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575386.588100:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575386.588103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575386.588105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575386.588111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575386.588122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575386.588128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff920a48353480 x1875444569957632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575386.588135:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569957632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575386.588140:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575386.588142:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575386.588144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575386.588147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575386.588158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575386.588163:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff920a48350380 x1875444569957760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575386.588170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569957760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575386.588175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575386.588177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575386.588179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575386.588182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575386.588192:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575386.588197:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575386/real 1788575386] req@ffff920a48350a80 x1875444569957888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575386.588205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569957888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575386.588210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575386.588212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575386.588214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569957888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575387.035837:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575387.611688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575388.059796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575388.635728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575389.083747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575389.659671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575390.107741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575390.683712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575391.131798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575391.707676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575391.707684:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575391.707689:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575391.707693:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575391.707702:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12145 00000100:00080000:1.0:1788575391.707711:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575391.707726:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575391.707756:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569958016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575391.707782:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575391.707784:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575391.707786:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575391.707789:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12145 00000100:00080000:1.0:1788575391.707792:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575391.707795:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575391.707804:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575391.707818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569958144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575391.707823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569958016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575391.707840:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575391.707844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575391.707846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575391.707852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12145 00000100:00080000:1.0:1788575391.707857:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575391.707862:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575391.707868:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575391.707881:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569958272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575391.707890:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575391.707894:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575391.707898:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575391.707900:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575391.707905:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12145 00000100:00080000:1.0:1788575391.707911:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575391.707916:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575391.707922:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575391.707929:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575391/real 1788575391] req@ffff920a52891f80 x1875444569958016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575391.707940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569958400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575391.707951:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569958016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575391.707954:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575391.707964:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575391.707967:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575391.707971:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575391.707973:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12145 00000100:00080000:1.0:1788575391.707978:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575391.707979:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575391.707984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575391.707987:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569958016:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788575391.707990:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575391.708003:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569958528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575391.708014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575391.708041:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575391.708054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575391/real 1788575391] req@ffff920a52890380 x1875444569958144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575391.708065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569958144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575391.708073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575391.708076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575391.708079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575391.708084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575391.708100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575391.708109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575391/real 1788575391] req@ffff920a52893b80 x1875444569958272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575391.708119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569958272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575391.708126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575391.708128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575391.708131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575391.708136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575391.708151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575391.708159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575391/real 1788575391] req@ffff920a52890000 x1875444569958400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575391.708168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569958400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575391.708180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575391.708183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575391.708186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575391.708199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575391.708214:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575391.708223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575391/real 1788575391] req@ffff920a52892680 x1875444569958528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575391.708267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569958528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575391.708274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575391.708277:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575391.708279:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575392.155740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575392.731699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575393.179850:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575393.755729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575394.203776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575394.779695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575395.227781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575395.803671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575396.251723:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575396.827669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575396.827689:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575396.827695:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575396.827699:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575396.827708:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12150 00000100:00080000:1.0:1788575396.827723:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575396.827729:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575396.827760:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569958656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575396.827780:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575396.827783:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575396.827784:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575396.827788:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12150 00000100:00080000:1.0:1788575396.827790:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575396.827794:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575396.827802:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575396.827812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569958656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575396.827814:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569958784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575396.827826:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575396.827828:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575396.827829:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575396.827833:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12150 00000100:00080000:1.0:1788575396.827835:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575396.827838:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575396.827843:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575396.827857:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569958912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575396.827867:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575396.827870:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575396.827874:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575396.827874:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575396.827884:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12150 00000100:00080000:1.0:1788575396.827890:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575396.827895:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575396.827901:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575396.827909:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575396/real 1788575396] req@ffff920a52890a80 x1875444569958656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575396.827921:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569959040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575396.827926:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569958656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575396.827945:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575396.827949:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575396.827951:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575396.827956:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575396.827959:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569958656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575396.827960:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575396.827970:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12150 00000100:00080000:1.0:1788575396.827973:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575396.827976:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575396.827980:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575396.827985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575396.827990:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569959168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575396.828008:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575396.828022:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575396/real 1788575396] req@ffff920a52892a00 x1875444569958784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575396.828030:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569958784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575396.828036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575396.828038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575396.828040:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575396.828043:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575396.828054:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575396.828060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575396/real 1788575396] req@ffff920a52892680 x1875444569958912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575396.828067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569958912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575396.828072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575396.828074:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575396.828076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569958912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575396.828080:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575396.828091:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575396.828096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575396/real 1788575396] req@ffff920a52890000 x1875444569959040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575396.828103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569959040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575396.828108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575396.828110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575396.828112:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575396.828120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575396.828130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575396.828136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575396/real 1788575396] req@ffff920a52893b80 x1875444569959168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575396.828142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569959168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575396.828147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575396.828148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575396.828150:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575397.275777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575397.851746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575398.299795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575398.875711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575399.323737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575399.899658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575400.347771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575400.923698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575401.371808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575401.947670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575401.947678:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575401.947689:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575401.947692:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575401.947702:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12155 00000100:00080000:2.0:1788575401.947711:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575401.947717:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575401.947747:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569959296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575401.947764:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575401.947767:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575401.947769:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575401.947773:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12155 00000100:00080000:2.0:1788575401.947777:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575401.947781:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575401.947789:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569959296:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575401.947791:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575401.947807:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569959424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575401.947818:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575401.947821:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575401.947823:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575401.947827:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12155 00000100:00080000:2.0:1788575401.947838:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575401.947842:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575401.947845:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575401.947852:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575401.947865:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569959552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575401.947870:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575401/real 1788575401] req@ffff920a60ac6a00 x1875444569959296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575401.947888:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575401.947892:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575401.947895:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575401.947901:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12155 00000100:00100000:3.0:1788575401.947901:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569959296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575401.947906:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575401.947914:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788575401.947920:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575401.947921:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575401.947927:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575401.947932:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569959296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575401.947936:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569959680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575401.947947:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575401.947950:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575401.947951:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575401.947955:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12155 00000100:00100000:3.0:1788575401.947958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959424:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575401.947962:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575401.947968:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575401.947973:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575401.947986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575401.947987:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569959808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575401.948003:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575401/real 1788575401] req@ffff920a60ac4700 x1875444569959424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575401.948018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569959424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575401.948026:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575401.948029:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575401.948032:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575401.948038:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575401.948055:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575401.948062:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575401/real 1788575401] req@ffff920a60ac4a80 x1875444569959552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575401.948070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569959552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575401.948074:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575401.948076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575401.948078:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575401.948081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575401.948091:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575401.948096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575401/real 1788575401] req@ffff920a60ac6300 x1875444569959680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575401.948103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569959680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575401.948107:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575401.948108:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575401.948110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575401.948147:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575401.948160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575401.948167:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575401/real 1788575401] req@ffff920a60ac5500 x1875444569959808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575401.948173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569959808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575426 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575401.948182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575401.948184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575401.948186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569959808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575402.395757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575402.971689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575403.419785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575403.995653:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575404.443723:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575405.019687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575405.467754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575406.043691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575406.491765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575407.067691:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.067701:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575407.067705:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575407.067705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575407.067719:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12160 00000100:00080000:2.0:1788575407.067728:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.067737:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575407.067772:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569959936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.067793:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.067796:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575407.067797:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.067800:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12160 00000100:00080000:2.0:1788575407.067802:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.067805:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575407.067813:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575407.067826:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569959936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575407.067827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569960064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.067841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.067845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575407.067847:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.067851:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12160 00000100:00080000:2.0:1788575407.067855:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.067859:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575407.067864:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575407.067878:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569960192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575407.067891:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.067894:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575407.067895:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.067899:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12160 00000100:00080000:2.0:1788575407.067902:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.067913:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575407.067919:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575407.067929:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569960320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575407.067934:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960064:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575407.067947:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575407.067950:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575407.067952:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575407.067957:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12160 00000100:00080000:2.0:1788575407.067962:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575407.067966:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575407.067971:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575407.067973:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575407.067993:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569960448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575407.068003:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575407.068029:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575407.515736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575408.091665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575408.539777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575409.115773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575409.563717:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575410.139707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788575410.139842:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788575410.139879:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575410.139894:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575410.139937:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575410.139952:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575410.139964:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575410.139978:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788575410.140060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575410] req@ffff920a60ac4a80 x1875444569960448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575410.140083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569960448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575410.140092:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575410.140097:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575410.140101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960448:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575410.140108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575410] req@ffff920a60ac6300 x1875444569960320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575410.140120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569960320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575410.140127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575410.140129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575410.140132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575410.140137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575410] req@ffff920a60ac7100 x1875444569960064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575410.140146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569960064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575410.140152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575410.140156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575410.140159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575410.140164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575410] req@ffff920a60ac5500 x1875444569960192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575410.140174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569960192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575410.140180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575410.140183:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575410.140186:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575410.140192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575407/real 1788575410] req@ffff920a60ac5180 x1875444569959936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575410.140202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569959936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575410.140210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575410.140213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575410.140216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569959936:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575410.587763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575411.163658:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575411.611755:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575412.187693:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575412.187693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575412.187710:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575412.187715:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.187725:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12165 00000100:00080000:2.0:1788575412.187735:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.187743:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575412.187772:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569960576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575412.187793:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.187796:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575412.187798:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.187801:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12165 00000100:00080000:2.0:1788575412.187803:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.187807:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575412.187817:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575412.187829:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569960576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575412.187832:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569960704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575412.187844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.187848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575412.187850:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.187854:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12165 00000100:00080000:2.0:1788575412.187858:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.187862:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575412.187867:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575412.187878:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569960832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575412.187888:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788575412.187891:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575412.187893:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575412.187896:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575412.187901:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12165 00000100:00080000:2.0:1788575412.187905:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.187910:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575412.187917:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575412.187928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff920a60ac4700 x1875444569960576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575412.187930:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569960960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575412.187950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569960576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575412.187979:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575412.187980:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575412.187987:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575412.187987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575412.187991:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575412.187995:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569960576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575412.187999:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12165 00000100:00080000:2.0:1788575412.188006:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575412.188010:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575412.188015:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575412.188036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575412.188042:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569961088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575412.188066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575412.188080:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff920a60ac6a00 x1875444569960704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575412.188093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569960704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575412.188102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575412.188105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575412.188108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575412.188114:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575412.188133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575412.188142:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff920a60ac4000 x1875444569960832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575412.188153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569960832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575412.188161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575412.188165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575412.188168:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575412.188173:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575412.188191:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575412.188200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff920a60ac5c00 x1875444569960960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575412.188212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569960960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575412.188226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575412.188229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575412.188232:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569960960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575412.188246:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575412.188264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575412.188273:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575412/real 1788575412] req@ffff920a60ac4380 x1875444569961088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575412.188285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569961088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575412.188292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575412.188295:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575412.188299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575412.635742:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575413.211660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575413.659756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575414.235684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575414.683715:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575415.259731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575415.707783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575416.283682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575416.731724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575417.307691:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575417.307701:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575417.307706:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575417.307706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575417.307730:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12170 00000100:00080000:2.0:1788575417.307740:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575417.307746:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575417.307778:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569961216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575417.307800:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575417.307803:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575417.307804:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575417.307808:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12170 00000100:00080000:2.0:1788575417.307812:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575417.307815:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575417.307823:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575417.307830:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569961216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575417.307836:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569961344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575417.307846:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575417.307848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575417.307849:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575417.307852:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12170 00000100:00080000:2.0:1788575417.307854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575417.307858:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575417.307861:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575417.307874:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569961472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575417.307882:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575417.307886:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575417.307888:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1788575417.307891:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575417.307896:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12170 00000100:00080000:2.0:1788575417.307901:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575417.307905:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575417.307915:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575417.307925:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569961600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575417.307934:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff920a60ac4e00 x1875444569961216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575417.307938:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575417.307942:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575417.307945:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575417.307952:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12170 00000100:00080000:2.0:1788575417.307956:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575417.307959:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575417.307962:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575417.307967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569961216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575417.307971:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569961728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575417.308001:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.308006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575417.308009:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569961216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575417.308028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.308046:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575417.308055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff920a60ac7800 x1875444569961344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.308063:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569961344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.308068:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.308070:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575417.308072:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575417.308076:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575417.308086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575417.308091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff920a60ac4380 x1875444569961472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.308102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569961472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.308107:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.308109:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575417.308110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575417.308113:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575417.308123:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575417.308128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff920a60ac5c00 x1875444569961600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.308137:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569961600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.308144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.308147:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575417.308149:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575417.308155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575417.308176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575417.308198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575417/real 1788575417] req@ffff920a60ac4000 x1875444569961728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575417.308210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569961728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575417.308218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575417.308221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575417.308227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575417.755795:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575418.331671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575418.779773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575419.355768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575419.803732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575420.379687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575420.827733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575421.403665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575421.851707:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575422.427684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575422.427693:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575422.427700:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575422.427703:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575422.427712:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12175 00000100:00080000:1.0:1788575422.427721:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575422.427728:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575422.427758:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569961856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575422.427778:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575422.427781:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575422.427782:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575422.427786:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12175 00000100:00080000:1.0:1788575422.427789:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575422.427792:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575422.427800:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575422.427807:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569961856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575422.427809:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569961984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575422.427825:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575422.427828:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575422.427828:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575422.427831:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12175 00000100:00080000:1.0:1788575422.427832:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575422.427834:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575422.427837:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575422.427843:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569962112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575422.427848:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575422.427850:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575422.427851:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575422.427854:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12175 00000100:00080000:1.0:1788575422.427856:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575422.427858:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575422.427861:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575422.427868:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569962240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575422.427876:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575422.427878:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575422.427880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575422.427882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575422.427885:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12175 00000100:00080000:1.0:1788575422.427887:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575422.427890:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575422.427892:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575422.427898:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569962368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575422.427912:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff920a52890380 x1875444569961856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575422.427940:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569961856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575422.427950:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575422.427953:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575422.427957:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569961856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575422.427983:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575422.428004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575422.428015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff920a52891f80 x1875444569961984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575422.428027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569961984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575422.428035:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575422.428038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575422.428041:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569961984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575422.428047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575422.428064:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575422.428074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff920a52891880 x1875444569962112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575422.428090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569962112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575422.428098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575422.428100:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575422.428104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575422.428109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575422.428125:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575422.428133:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff920a52893480 x1875444569962240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575422.428143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569962240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575422.428150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575422.428153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575422.428156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575422.428161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575422.428175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575422.428183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575422/real 1788575422] req@ffff920a52891500 x1875444569962368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575422.428193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569962368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575422.428205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575422.428208:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575422.428211:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575422.875760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575423.451699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575423.899757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575424.475749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575424.923785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575425.499675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575425.947746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575426.523673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575426.971770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575427.547683:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575427.547690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575427.547691:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575427.547695:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575427.547703:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12180 00000100:00080000:0.0:1788575427.547712:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575427.547718:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575427.547748:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569962496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575427.547767:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575427.547769:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575427.547770:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575427.547773:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12180 00000100:00080000:0.0:1788575427.547775:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575427.547778:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575427.547785:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575427.547793:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569962624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575427.547799:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569962496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575427.547803:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575427.547806:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575427.547807:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575427.547810:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12180 00000100:00080000:0.0:1788575427.547812:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575427.547813:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575427.547816:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575427.547823:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569962752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575427.547828:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575427.547830:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575427.547831:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575427.547834:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12180 00000100:00080000:0.0:1788575427.547836:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575427.547838:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575427.547842:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575427.547849:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569962880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575427.547854:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575427.547861:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575427.547864:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575427.547865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575427.547868:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12180 00000100:00080000:0.0:1788575427.547872:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575427.547874:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575427.547886:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575427.547894:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569963008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575427.547898:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff920a48353800 x1875444569962496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575427.547914:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569962496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575427.547922:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575427.547925:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575427.547927:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569962496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575427.547945:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575427.547961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575427.547969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff920a48350700 x1875444569962624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575427.547976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569962624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575427.547985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575427.547987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575427.547989:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575427.547992:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575427.548001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575427.548006:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff920a48353b80 x1875444569962752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575427.548012:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569962752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575427.548017:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575427.548018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575427.548020:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575427.548023:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575427.548033:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575427.548038:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff920a48350a80 x1875444569962880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575427.548044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569962880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575427.548048:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575427.548050:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575427.548052:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569962880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575427.548055:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575427.548064:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575427.548069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575427/real 1788575427] req@ffff920a48350380 x1875444569963008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575427.548074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569963008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575427.548079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575427.548080:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575427.548082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575427.995742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575428.571675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575429.019709:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575429.595701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575430.043750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575430.619676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575431.067729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575431.643699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575432.091734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575432.667677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575432.667701:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575432.667708:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575432.667713:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575432.667722:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12186 00000100:00080000:1.0:1788575432.667732:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575432.667738:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575432.667772:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569963136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575432.667799:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575432.667803:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575432.667805:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575432.667809:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12186 00000100:00080000:1.0:1788575432.667813:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575432.667818:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575432.667828:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575432.667847:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569963136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575432.667856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569963264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575432.667867:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575432.667871:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575432.667873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575432.667877:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12186 00000100:00080000:1.0:1788575432.667882:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575432.667886:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575432.667892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788575432.667904:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575432.667905:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569963392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575432.667917:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575432.667921:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575432.667923:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575432.667928:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12186 00000100:00080000:1.0:1788575432.667933:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575432.667938:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575432/real 1788575432] req@ffff920a52890e00 x1875444569963136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575432.667939:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575432.667949:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575432.667957:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569963136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575432.667964:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569963520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575432.667968:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575432.667977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575432.667980:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575432.667984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569963136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575432.667986:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575432.667991:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575432.667998:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12186 00000100:00080000:1.0:1788575432.668002:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575432.668006:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575432.668012:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575432.668016:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575432.668027:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569963648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575432.668038:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575432.668050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575432/real 1788575432] req@ffff920a52893800 x1875444569963264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575432.668058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569963264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575432.668064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575432.668066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575432.668068:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575432.668071:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575432.668081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575432.668086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575432/real 1788575432] req@ffff920a52892d80 x1875444569963392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575432.668093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569963392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575432.668097:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575432.668099:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575432.668100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575432.668103:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575432.668112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575432.668117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575432/real 1788575432] req@ffff920a52892300 x1875444569963520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575432.668124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569963520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575432.668129:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575432.668130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575432.668132:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575432.668140:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575432.668155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575432.668164:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575432/real 1788575432] req@ffff920a52891180 x1875444569963648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575432.668175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569963648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575432.668183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575432.668186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575432.668189:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575433.115747:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575433.691688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575434.139735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575434.715712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575435.163743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575435.739722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575436.187829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575436.763695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575437.211768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575437.787712:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575437.787712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575437.787723:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575437.787728:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575437.787739:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12191 00000100:00080000:0.0:1788575437.787750:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575437.787757:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575437.787815:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569963776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575437.787843:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575437.787847:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575437.787853:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575437.787859:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12191 00000100:00080000:0.0:1788575437.787864:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575437.787870:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575437.787884:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575437.787895:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569963776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575437.787899:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569963904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575437.787909:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575437.787913:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575437.787915:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575437.787918:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12191 00000100:00080000:0.0:1788575437.787921:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575437.787924:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575437.787927:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575437.787938:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569964032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575437.787945:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575437.787956:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575437.787957:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575437.787964:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12191 00000100:00080000:0.0:1788575437.787969:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575437.787974:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788575437.787978:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575437.787985:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575437.788002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569964160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575437.788014:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575437.788018:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575437/real 1788575437] req@ffff920a48353480 x1875444569963776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575437.788023:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575437.788029:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575437.788037:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12191 00000100:00080000:0.0:1788575437.788042:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575437.788042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569963776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575437.788052:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575437.788065:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575437.788067:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575437.788074:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575437.788078:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569963776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575437.788085:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569964288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575437.788102:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575437.788122:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575437.788132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575437/real 1788575437] req@ffff920a48351c00 x1875444569963904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575437.788140:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569963904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575437.788146:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575437.788149:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575437.788151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569963904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575437.788154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575437.788165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575437.788171:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575437/real 1788575437] req@ffff920a48351500 x1875444569964032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575437.788178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569964032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575437.788185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575437.788189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575437.788193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575437.788199:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575437.788220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575437.788232:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575437/real 1788575437] req@ffff920a48350380 x1875444569964160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575437.788245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569964160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575437.788252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575437.788254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575437.788257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575437.788261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575437.788277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575437.788284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575437/real 1788575437] req@ffff920a48350a80 x1875444569964288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575437.788291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569964288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575437.788296:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575437.788298:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575437.788300:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575438.235816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575438.811731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575439.259817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575439.835744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575440.283811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575440.859733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575441.307798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575441.883700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575442.331794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575442.907661:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575442.907661:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575442.907674:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575442.907680:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575442.907694:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12196 00000100:00080000:0.0:1788575442.907706:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575442.907715:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575442.907806:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569964416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575442.907842:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575442.907847:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575442.907849:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575442.907854:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12196 00000100:00080000:0.0:1788575442.907859:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575442.907864:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575442.907876:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575442.907892:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569964544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575442.907906:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575442.907910:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575442.907914:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575442.907916:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569964416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575442.907922:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12196 00000100:00080000:0.0:1788575442.907931:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575442.907936:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575442.907941:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575442.907955:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569964672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575442.907965:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575442.907969:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575442.907971:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575442.907975:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12196 00000100:00080000:0.0:1788575442.907980:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575442.907985:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575442.907991:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575442.908006:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569964800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575442.908016:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575442.908021:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1788575442.908024:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575442.908025:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575442.908033:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12196 00000100:00080000:0.0:1788575442.908038:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575442.908043:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575442.908049:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575442.908062:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569964928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575442.908077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575442/real 1788575442] req@ffff920a48353b80 x1875444569964416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575442.908113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569964416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575442.908127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575442.908133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575442.908138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569964416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575442.908168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575442.908195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575442.908211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575442/real 1788575442] req@ffff920a48350700 x1875444569964544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575442.908237:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569964544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575442.908247:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575442.908251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575442.908255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575442.908262:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575442.908286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575442.908299:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575442/real 1788575442] req@ffff920a48353800 x1875444569964672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575442.908324:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569964672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575442.908340:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575442.908345:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575442.908349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575442.908356:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575442.908379:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575442.908391:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575442/real 1788575442] req@ffff920a48350a80 x1875444569964800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575442.908404:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569964800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575442.908413:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575442.908417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575442.908421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575442.908428:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575442.908450:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575442.908474:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575442/real 1788575442] req@ffff920a48350380 x1875444569964928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575442.908489:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569964928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575442.908498:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575442.908501:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575442.908505:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569964928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575443.355777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575443.931722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575444.379765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575444.955711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575445.403814:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575445.979734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575446.427775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575447.003736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575447.451793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575448.027657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575448.027671:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575448.027679:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575448.027683:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575448.027705:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12201 00000100:00080000:0.0:1788575448.027714:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575448.027722:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575448.027755:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569965056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575448.027786:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575448.027790:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575448.027792:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575448.027797:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12201 00000100:00080000:0.0:1788575448.027801:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575448.027805:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575448.027814:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575448.027827:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569965184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575448.027839:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575448.027842:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569965056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575448.027843:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575448.027846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575448.027852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12201 00000100:00080000:0.0:1788575448.027857:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575448.027861:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575448.027867:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575448.027878:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569965312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575448.027887:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575448.027890:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575448.027892:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575448.027896:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12201 00000100:00080000:0.0:1788575448.027899:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575448.027902:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788575448.027907:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575448.027908:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575448.027924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569965440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575448.027933:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575448.027937:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575448.027938:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff920a48351500 x1875444569965056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.027940:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575448.027949:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12201 00000100:00080000:0.0:1788575448.027956:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575448.027959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569965056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575448.027962:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575448.027973:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575448.027975:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575448.027980:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575448.027983:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569965056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575448.027988:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569965568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575448.028004:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575448.028021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575448.028028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff920a48351c00 x1875444569965184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575448.028035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569965184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575448.028040:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575448.028042:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575448.028044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575448.028047:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575448.028057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575448.028069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff920a48353480 x1875444569965312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575448.028077:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569965312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575448.028082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575448.028083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575448.028085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575448.028088:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575448.028099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575448.028104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff920a48350380 x1875444569965440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575448.028110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569965440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575448.028115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575448.028116:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575448.028118:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575448.028120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575448.028129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575448.028134:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575448/real 1788575448] req@ffff920a48350a80 x1875444569965568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575448.028142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569965568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575448.028149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575448.028152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575448.028155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575448.475694:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575449.051700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575449.499853:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575450.075665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575450.523845:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575451.099649:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575451.547781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575452.123683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575452.571743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575453.147695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575453.147729:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575453.147733:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575453.147736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575453.147744:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12206 00000100:00080000:3.0:1788575453.147753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575453.147758:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575453.147793:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569965696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575453.147819:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575453.147823:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575453.147824:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575453.147829:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12206 00000100:00080000:3.0:1788575453.147833:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575453.147837:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575453.147845:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569965696:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575453.147847:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575453.147862:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569965824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575453.147870:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575453.147873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575453.147875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575453.147878:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12206 00000100:00080000:3.0:1788575453.147881:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575453.147883:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575453.147886:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575453.147893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569965952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575453.147901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575453.147901:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575453.147906:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575453.147907:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575453.147911:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12206 00000100:00080000:3.0:1788575453.147915:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575453.147918:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575453.147923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575453.147929:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff920a529b3100 x1875444569965696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575453.147937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569966080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575453.147945:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569965696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.147952:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.147955:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575453.147958:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569965696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575453.147974:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575453.147987:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575453.147996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff920a529b1c00 x1875444569965824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.148004:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569965824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.148009:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.148013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575453.148019:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965824:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575453.148023:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575453.148027:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965952:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575453.148029:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575453.148032:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575453.148036:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12206 00000100:00080000:3.0:1788575453.148040:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575453.148043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575453.148047:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575453.148052:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575453.148053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff920a529b2a00 x1875444569965952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.148062:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569965952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575453.148066:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569966208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575453.148070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.148073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575453.148075:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569965952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575453.148082:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575453.148093:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575453.148100:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff920a529b3480 x1875444569966080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.148106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569966080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.148111:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.148112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575453.148114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575453.148208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575453.148231:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575453.148243:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575453/real 1788575453] req@ffff920a529b2d80 x1875444569966208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575453.148252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569966208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575453.148257:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575453.148259:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575453.148260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575453.595798:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575454.171720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575454.619766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575455.195713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575455.643789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575456.219750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575456.667719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575457.243723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575457.691802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575458.267701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575458.267707:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.267721:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575458.267727:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.267740:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12211 00000100:00080000:1.0:1788575458.267753:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.267762:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575458.267808:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569966336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.267839:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.267843:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575458.267846:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.267851:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12211 00000100:00080000:1.0:1788575458.267855:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.267859:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575458.267871:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575458.267876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569966336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575458.267887:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569966464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.267898:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575458.267901:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575458.267903:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.267907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12211 00000100:00080000:1.0:1788575458.267911:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.267915:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575458.267920:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575458.267934:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569966592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.267943:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1788575458.267944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575458.267950:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575458.267952:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575458.267957:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12211 00000100:00080000:1.0:1788575458.267962:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575458.267967:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575458.267973:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575458.267980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575458] req@ffff920a52893100 x1875444569966336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575458.267987:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569966720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575458.268000:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575458.268000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569966336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575458.268008:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575458.268013:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575458.268017:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575458.268021:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12211 00000100:00080000:3.0:1788575458.268024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:1.0:1788575458.268028:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575458.268029:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569966336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575458.268037:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575458.268043:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575458.268053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575458.268056:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569966848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575458.268072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575458.268083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575458] req@ffff920a52890700 x1875444569966464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575458.268091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569966464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575458.268096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575458.268099:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575458.268101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575458.268104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575458.268114:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575458.268119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575458] req@ffff920a52891180 x1875444569966592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575458.268126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569966592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575458.268130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575458.268132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575458.268133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575458.268136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575458.268145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575458.268150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575458] req@ffff920a52892300 x1875444569966720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575458.268156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569966720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575458.268159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575458.268161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575458.268163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575458.268171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575458.268179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575458.268184:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575458/real 1788575458] req@ffff920a52892d80 x1875444569966848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575458.268195:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569966848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575458.268199:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575458.268201:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575458.268202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569966848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575458.715763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575459.291724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575459.739771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575460.315763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575460.763798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575461.339704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575461.787775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575462.363708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575462.811749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575463.387704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575463.387745:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.387753:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575463.387757:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.387774:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12216 00000100:00080000:2.0:1788575463.387784:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.387791:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575463.387832:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569966976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.387858:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.387861:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.387862:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.387865:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12216 00000100:00080000:2.0:1788575463.387868:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.387870:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575463.387880:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.387890:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569967104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.387898:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.387905:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:3.0:1788575463.387906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569966976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575463.387908:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.387918:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12216 00000100:00080000:2.0:1788575463.387926:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.387934:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575463.387940:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575463.387955:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569967232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.387964:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.387968:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575463.387971:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575463.387974:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12216 00000100:00080000:2.0:1788575463.387980:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.387985:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:3.0:1788575463.387992:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575463.387997:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575463.388011:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569967360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575463.388019:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575463.388021:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575463.388023:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575463.388029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff920a60ac6a00 x1875444569966976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575463.388032:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12216 00000100:00080000:2.0:1788575463.388041:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575463.388053:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575463.388062:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575463.388068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569966976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575463.388076:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569967488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575463.388080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575463.388093:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575463.388104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569966976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575463.388128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575463.388144:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575463.388152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff920a60ac4700 x1875444569967104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575463.388164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569967104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575463.388170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575463.388171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575463.388173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575463.388177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575463.388188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575463.388193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff920a60ac5180 x1875444569967232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575463.388200:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569967232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575463.388205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575463.388207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575463.388208:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575463.388212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575463.388221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575463.388227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff920a60ac5500 x1875444569967360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575463.388233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569967360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575463.388241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575463.388243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575463.388245:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575463.388248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575463.388257:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575463.388262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575463/real 1788575463] req@ffff920a60ac7100 x1875444569967488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575463.388268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569967488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575463.388272:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575463.388273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575463.388275:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575463.835743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575464.411919:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575464.859805:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575465.435698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575465.883763:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575466.459700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575466.907741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575467.483704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575467.931757:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575468.507701:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575468.507710:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575468.507713:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575468.507723:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12221 00000100:00100000:0.0:1788575468.507724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575468.507733:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575468.507740:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575468.507771:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569967616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575468.507791:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575468.507794:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575468.507795:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575468.507797:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12221 00000100:00080000:2.0:1788575468.507799:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575468.507801:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575468.507808:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575468.507815:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569967744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575468.507821:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575468.507824:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575468.507825:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575468.507827:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12221 00000100:00080000:2.0:1788575468.507829:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575468.507834:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575468.507836:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575468.507843:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569967872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575468.507850:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575468.507852:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575468.507854:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575468.507877:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12221 00000100:00080000:2.0:1788575468.507882:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1788575468.507882:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569967616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575468.507890:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575468.507899:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575468.507913:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569968000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575468.507922:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575468.507925:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575468.507927:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575468.507930:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12221 00000100:00080000:2.0:1788575468.507933:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575468.507937:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575468.507941:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575468.507952:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569968128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575468.507977:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575468.508031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff920a60ac6300 x1875444569967616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575468.508052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569967616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575468.508064:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575468.508067:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575468.508070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569967616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575468.508075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575468.508098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575468.508111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff920a60ac4a80 x1875444569967744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575468.508122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569967744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575468.508130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575468.508134:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575468.508137:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575468.508166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575468.508181:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575468.508188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff920a60ac7100 x1875444569967872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575468.508195:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569967872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575468.508200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575468.508202:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575468.508204:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569967872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575468.508209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575468.508219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575468.508224:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff920a60ac5500 x1875444569968000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575468.508231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569968000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575468.508236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575468.508237:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575468.508239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575468.508242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575468.508250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575468.508255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575468/real 1788575468] req@ffff920a60ac5180 x1875444569968128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575468.508262:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569968128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575468.508266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575468.508267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575468.508269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575468.955791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575469.531649:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575469.979725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575470.555712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575471.003763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575471.579739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575472.027732:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575472.603719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575473.051754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575473.627680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575473.627701:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575473.627706:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575473.627710:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575473.627718:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12227 00000100:00080000:2.0:1788575473.627728:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575473.627734:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575473.627768:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569968256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575473.627787:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575473.627790:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575473.627792:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575473.627796:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12227 00000100:00080000:2.0:1788575473.627798:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575473.627801:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575473.627811:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575473.627812:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569968256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575473.627822:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569968384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575473.627831:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575473.627833:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575473.627835:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575473.627838:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12227 00000100:00080000:2.0:1788575473.627840:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575473.627844:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575473.627850:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575473.627862:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569968512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575473.627872:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575473.627876:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575473.627878:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575473.627883:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12227 00000100:00080000:2.0:1788575473.627888:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575473.627893:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575473.627900:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575473.627901:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575473.627919:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569968640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575473.627929:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575473.627932:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575473.627933:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575473.627937:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968512:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575473.627940:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12227 00000100:00080000:2.0:1788575473.627956:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575473.627960:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575473.627963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968640:192.168.202.157@tcp:8:kworker.0 00000008:00100000:2.0:1788575473.627967:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575473.627979:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569968768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575473.627995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575474.075779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575474.651712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575475.099814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575475.675704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575476.123795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575476.699708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575476.699891:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575476.699948:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575476.699965:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575476.700017:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575476.700031:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575476.700043:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788575476.700054:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575473/real 1788575476] req@ffff920a60ac6a00 x1875444569968384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1788575476.700055:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:3.0:1788575476.700084:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569968384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575476.700113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575476.700119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575476.700124:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575476.700132:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575473/real 1788575476] req@ffff920a60ac4000 x1875444569968512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575476.700144:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569968512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575476.700152:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575476.700155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575476.700159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575476.700165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575473/real 1788575476] req@ffff920a60ac4700 x1875444569968256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575476.700177:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569968256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575476.700185:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575476.700188:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575476.700192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569968256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575476.700216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575473/real 1788575476] req@ffff920a60ac4380 x1875444569968768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575476.700227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569968768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575476.700235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575476.700238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575476.700241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968768:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575476.700248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575473/real 1788575476] req@ffff920a60ac5c00 x1875444569968640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575476.700259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569968640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575476.700266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575476.700269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575476.700272:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569968640:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575477.147766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575477.723716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575478.171762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575478.747689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575478.747747:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575478.747755:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575478.747759:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575478.747767:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12232 00000100:00080000:1.0:1788575478.747777:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575478.747784:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575478.747818:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569968896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575478.747839:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575478.747842:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575478.747843:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575478.747847:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12232 00000100:00080000:1.0:1788575478.747850:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575478.747855:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575478.747867:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575478.747878:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569968896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575478.747883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569969024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575478.747895:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575478.747898:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575478.747900:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575478.747904:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12232 00000100:00080000:1.0:1788575478.747907:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575478.747911:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575478.747916:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575478.747937:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569969152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575478.747951:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575478.747955:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575478.747957:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575478.747960:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12232 00000100:00080000:1.0:1788575478.747964:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575478.747969:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575478.747974:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575478.747985:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569969280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575478.747990:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575478.747999:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575478.748003:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575478.748005:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575478.748009:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12232 00000100:00080000:1.0:1788575478.748013:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575478.748018:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575478.748023:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575478.748037:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569969408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575478.748046:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575478/real 1788575478] req@ffff920a52893800 x1875444569968896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575478.748074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569968896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575478.748087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575478.748092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575478.748097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569968896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575478.748129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575478.748186:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575478.748200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575478/real 1788575478] req@ffff920a52890e00 x1875444569969024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575478.748211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569969024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575478.748218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575478.748220:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575478.748223:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575478.748228:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575478.748249:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575478.748259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575478/real 1788575478] req@ffff920a52891500 x1875444569969152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575478.748268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569969152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575478.748280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575478.748284:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575478.748286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575478.748291:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575478.748311:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575478.748319:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575478/real 1788575478] req@ffff920a52893480 x1875444569969280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575478.748330:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569969280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575478.748338:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575478.748341:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575478.748344:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575478.748349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575478.748367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575478.748378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575478/real 1788575478] req@ffff920a52891880 x1875444569969408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575478.748386:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569969408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575478.748393:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575478.748396:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575478.748400:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575479.195749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575479.771664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575480.219727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575480.795721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575481.243847:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575481.819659:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575482.267722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575482.843699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575483.291768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575483.867690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575483.867694:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575483.867702:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575483.867706:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575483.867715:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12237 00000100:00080000:1.0:1788575483.867726:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575483.867735:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575483.867785:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569969536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575483.867809:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575483.867813:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575483.867815:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575483.867820:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12237 00000100:00080000:1.0:1788575483.867825:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575483.867829:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575483.867839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569969536:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788575483.867840:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575483.867857:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569969664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575483.867869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575483.867872:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575483.867874:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575483.867879:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12237 00000100:00080000:1.0:1788575483.867883:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575483.867887:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575483.867893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1788575483.867895:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575483.867909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569969792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575483.867920:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575483.867924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575483.867926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575483.867930:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12237 00000100:00100000:3.0:1788575483.867935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575483/real 1788575483] req@ffff920a52891f80 x1875444569969536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575483.867936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575483.867946:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575483.867953:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575483.867955:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569969536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575483.867964:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575483.867967:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575483.867969:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569969536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575483.867979:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569969920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575483.867988:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969664:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788575483.867992:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575483.867997:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575483.867999:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575483.868006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12237 00000800:00000100:3.0:1788575483.868007:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575483.868014:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575483.868022:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575483.868023:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575483/real 1788575483] req@ffff920a52890380 x1875444569969664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575483.868031:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575483.868034:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569969664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575483.868041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575483.868043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575483.868048:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569970048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575483.868048:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575483.868057:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575483.868076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575483.868083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575483/real 1788575483] req@ffff920a52893b80 x1875444569969792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575483.868090:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569969792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575483.868095:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575483.868097:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575483.868099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575483.868106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575483.868116:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575483.868121:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575483/real 1788575483] req@ffff920a52890000 x1875444569969920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575483.868128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569969920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575483.868132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575483.868133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575483.868135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569969920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575483.868138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575483.868146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575483.868151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575483/real 1788575483] req@ffff920a52892680 x1875444569970048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575483.868157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569970048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575483.868162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575483.868164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575483.868165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575484.315753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575484.891676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575485.339723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575485.915677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575486.363762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575486.939690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575487.388108:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575487.963693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575488.411781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575488.987670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575488.987674:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575488.987681:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575488.987684:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575488.987693:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12242 00000100:00080000:2.0:1788575488.987702:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575488.987708:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575488.987742:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569970176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575488.987766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575488.987770:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575488.987772:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575488.987776:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12242 00000100:00080000:2.0:1788575488.987780:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575488.987792:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575488.987793:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569970176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575488.987802:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575488.987816:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569970304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575488.987826:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575488.987829:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575488.987832:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575488.987836:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12242 00000100:00080000:2.0:1788575488.987840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575488.987844:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575488.987850:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575488.987859:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575488.987864:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569970432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575488.987873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575488.987877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575488.987879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575488.987882:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12242 00000100:00080000:2.0:1788575488.987886:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575488.987891:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575488.987896:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575488/real 1788575488] req@ffff920a60ac7800 x1875444569970176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575488.987899:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575488.987912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569970560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575488.987920:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569970176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575488.987925:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575488.987933:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575488.987938:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575488.987941:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575488.987945:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12242 00000100:00080000:1.0:1788575488.987950:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575488.987951:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575488.987957:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569970176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575488.987958:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575488.987967:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575488.987979:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569970688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575488.987983:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575488.988014:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575488.988028:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575488/real 1788575488] req@ffff920a60ac4e00 x1875444569970304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575488.988049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569970304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575488.988058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575488.988062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575488.988065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575488.988072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575488.988112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575488.988128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575488/real 1788575488] req@ffff920a60ac5c00 x1875444569970432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575488.988141:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569970432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575488.988151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575488.988155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575488.988159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575488.988166:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575488.988188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575488.988210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575488/real 1788575488] req@ffff920a60ac4380 x1875444569970560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575488.988222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569970560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575488.988235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575488.988239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575488.988242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575488.988256:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575488.988273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575488.988282:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575488/real 1788575488] req@ffff920a60ac4700 x1875444569970688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575488.988294:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569970688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575513 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575488.988302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575488.988306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575488.988311:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575489.435774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575490.011715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575490.459819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575491.035697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575491.483747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575492.059769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575492.507794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575493.083722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575493.531759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575494.107659:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575494.107669:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575494.107673:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575494.107678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575494.107691:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12247 00000100:00080000:2.0:1788575494.107706:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575494.107713:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575494.107744:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569970816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575494.107772:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575494.107776:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575494.107778:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575494.107783:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12247 00000100:00080000:2.0:1788575494.107786:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575494.107789:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575494.107799:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575494.107811:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569970944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575494.107822:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575494.107826:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569970816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575494.107827:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575494.107830:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575494.107839:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12247 00000100:00080000:2.0:1788575494.107843:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575494.107846:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575494.107851:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575494.107863:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569971072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575494.107872:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575494.107875:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575494.107877:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575494.107881:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12247 00000100:00080000:2.0:1788575494.107884:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575494.107890:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575494.107891:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575494.107899:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575494.107912:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569971200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575494.107922:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575494.107926:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575494.107929:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575494.107930:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff920a60ac4000 x1875444569970816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575494.107935:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12247 00000100:00080000:2.0:1788575494.107940:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575494.107944:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575494.107949:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575494.107963:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569970816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575494.107964:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569971328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575494.107992:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575494.107998:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575494.108003:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569970816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575494.108023:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575494.108045:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575494.108056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff920a60ac6a00 x1875444569970944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.108067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569970944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575494.108075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575494.108078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575494.108081:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569970944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575494.108087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575494.108104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575494.108114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff920a60ac5180 x1875444569971072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.108125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569971072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575494.108132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575494.108135:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575494.108138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575494.108145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575494.108160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575494.108168:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff920a60ac5500 x1875444569971200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.108179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569971200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575494.108188:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575494.108190:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575494.108193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575494.108198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575494.108212:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575494.108221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575494/real 1788575494] req@ffff920a60ac7100 x1875444569971328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575494.108231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569971328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575494.108238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575494.108241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575494.108244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575494.555785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575495.131695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575495.579807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575496.155709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575496.603817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575497.179727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575497.627780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575498.203779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575498.651841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575499.227680:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575499.227691:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575499.227695:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575499.227706:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12252 00000100:00100000:0.0:1788575499.227716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575499.227716:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575499.227725:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575499.227789:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569971456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575499.227816:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575499.227820:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575499.227822:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575499.227826:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12252 00000100:00080000:2.0:1788575499.227831:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575499.227835:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575499.227843:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575499.227854:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569971584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575499.227861:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575499.227863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575499.227864:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575499.227870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569971456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575499.227870:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12252 00000100:00080000:2.0:1788575499.227873:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575499.227877:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575499.227881:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575499.227888:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569971712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575499.227894:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575499.227896:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575499.227897:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575499.227899:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12252 00000100:00080000:2.0:1788575499.227901:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575499.227902:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575499.227905:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575499.227911:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569971840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575499.227922:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575499.227925:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575499.227926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575499.227932:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12252 00000800:00000100:1.0:1788575499.227934:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575499.227937:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575499.227942:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575499.227946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575499.227957:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569971968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575499.227970:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff920a60ac4a80 x1875444569971456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575499.227989:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569971456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.227999:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575499.228003:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575499.228007:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569971456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575499.228028:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575499.228051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575499.228064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff920a60ac6300 x1875444569971584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575499.228075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569971584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.228083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575499.228086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575499.228089:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575499.228095:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575499.228112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575499.228122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff920a60ac7100 x1875444569971712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575499.228134:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569971712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.228141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575499.228145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575499.228148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575499.228153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575499.228172:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575499.228182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff920a60ac5500 x1875444569971840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575499.228193:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569971840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.228200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575499.228204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575499.228207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575499.228212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575499.228227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575499.228236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575499/real 1788575499] req@ffff920a60ac5180 x1875444569971968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575499.228247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569971968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575499.228260:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575499.228263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575499.228267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569971968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575499.675790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575500.251732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575500.699735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575501.275672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575501.723777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575502.299694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575502.747775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575503.323663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575503.771774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575504.347707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575504.347732:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575504.347742:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575504.347746:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575504.347756:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12257 00000100:00080000:2.0:1788575504.347766:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575504.347774:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575504.347805:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569972096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575504.347831:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575504.347835:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575504.347837:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575504.347842:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12257 00000100:00080000:2.0:1788575504.347847:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575504.347851:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575504.347861:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575504.347868:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569972096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575504.347875:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569972224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575504.347901:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575504.347905:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575504.347907:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575504.347912:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12257 00000100:00080000:2.0:1788575504.347918:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575504.347922:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575504.347928:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575504.347942:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569972352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575504.347952:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575504.347955:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575504.347957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575504.347960:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12257 00000100:00080000:2.0:1788575504.347964:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575504.347971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575504.347971:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575504.347982:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575504.347998:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569972480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575504.348010:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575504.348014:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff920a60ac6a00 x1875444569972096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575504.348016:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575504.348021:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575504.348033:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12257 00000100:00080000:2.0:1788575504.348038:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575504.348040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569972096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.348051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.348058:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575504.348059:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575504.348063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569972096:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575504.348074:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575504.348090:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569972608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575504.348098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575504.348133:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575504.348149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff920a60ac4000 x1875444569972224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.348162:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569972224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.348171:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.348174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575504.348178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575504.348185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575504.348211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575504.348219:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff920a60ac4700 x1875444569972352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.348226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569972352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.348231:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.348233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575504.348235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575504.348239:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575504.348251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575504.348256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff920a60ac4380 x1875444569972480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.348263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569972480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.348267:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.348268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575504.348270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575504.348280:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575504.348293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575504.348302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575504/real 1788575504] req@ffff920a60ac5c00 x1875444569972608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575504.348313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569972608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575504.348321:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575504.348324:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575504.348327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575504.795734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575505.371716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575505.819812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575506.395719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575506.843697:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575507.419738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575507.867739:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575508.443674:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575508.891782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575509.467697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575509.467794:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575509.467798:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575509.467802:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575509.467810:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12262 00000100:00080000:2.0:1788575509.467819:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575509.467825:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575509.467855:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569972736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575509.467873:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575509.467875:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575509.467877:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575509.467880:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12262 00000100:00080000:2.0:1788575509.467882:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575509.467885:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575509.467892:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575509.467897:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569972736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575509.467903:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569972864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575509.467915:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575509.467917:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575509.467919:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575509.467921:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12262 00000100:00080000:2.0:1788575509.467923:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575509.467925:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575509.467927:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575509.467935:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569972992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575509.467941:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575509.467943:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575509.467945:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575509.467947:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12262 00000100:00080000:2.0:1788575509.467950:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575509.467953:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575509.467958:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788575509.467971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575509.467971:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569973120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575509.467982:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575509.467985:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575509.467987:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575509.467991:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12262 00000100:00080000:2.0:1788575509.467996:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575509.468002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff920a60ac4e00 x1875444569972736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575509.468002:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575509.468009:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575509.468021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569972736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575509.468022:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569973248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575509.468055:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.468059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575509.468063:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569972736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575509.468081:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575509.468098:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575509.468106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff920a60ac7800 x1875444569972864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.468113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569972864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.468118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.468121:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575509.468122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575509.468125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575509.468135:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575509.468139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff920a60ac5c00 x1875444569972992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.468146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569972992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.468150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.468152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575509.468153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569972992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575509.468157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575509.468166:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575509.468180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff920a60ac4380 x1875444569973120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.468187:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569973120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.468192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.468193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575509.468195:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575509.468198:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575509.468208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575509.468212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575509/real 1788575509] req@ffff920a60ac4700 x1875444569973248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575509.468219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569973248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575509.468223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575509.468224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575509.468226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575509.915759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575510.491689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575510.939722:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575511.515683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575511.963745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575512.539688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575512.987764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575513.563708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575514.011747:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575514.587698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575514.587711:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575514.587718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575514.587722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575514.587732:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12267 00000100:00080000:0.0:1788575514.587742:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575514.587750:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575514.587787:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569973376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575514.587814:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575514.587818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575514.587820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575514.587825:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12267 00000100:00080000:0.0:1788575514.587829:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575514.587833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575514.587840:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569973376:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575514.587843:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575514.587859:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569973504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575514.587870:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575514.587873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575514.587875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575514.587879:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12267 00000100:00080000:0.0:1788575514.587884:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575514.587889:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575514.587893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1788575514.587894:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575514.587909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569973632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575514.587919:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575514.587923:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1788575514.587926:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff920a48353800 x1875444569973376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575514.587927:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575514.587935:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12267 00000100:00080000:0.0:1788575514.587942:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575514.587947:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575514.587950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569973376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575514.587954:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575514.587967:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575514.587973:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569973760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575514.587976:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575514.587984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569973376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575514.587990:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575514.587994:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575514.587996:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575514.588001:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12267 00000100:00080000:0.0:1788575514.588004:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575514.588008:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575514.588012:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973504:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788575514.588015:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575514.588031:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569973888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575514.588043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575514.588059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff920a48350700 x1875444569973504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575514.588073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569973504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575514.588081:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575514.588085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575514.588088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575514.588094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575514.588112:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575514.588122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff920a48353b80 x1875444569973632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575514.588132:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569973632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575514.588141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575514.588143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575514.588145:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575514.588148:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575514.588160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575514.588166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff920a48350a80 x1875444569973760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575514.588172:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569973760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575514.588177:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575514.588178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575514.588194:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575514.588209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575514.588229:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575514.588240:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575514/real 1788575514] req@ffff920a48350380 x1875444569973888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575514.588250:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569973888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575514.588258:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575514.588261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575514.588264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569973888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575515.035768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575515.611688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575516.059760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575516.635666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575517.083724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575517.659729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575518.107833:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575518.683673:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575519.131718:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575519.707703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575519.707705:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575519.707713:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575519.707717:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575519.707726:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12273 00000100:00080000:3.0:1788575519.707736:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575519.707742:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575519.707773:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569974016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575519.707793:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575519.707795:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575519.707796:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575519.707799:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12273 00000100:00080000:3.0:1788575519.707801:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575519.707803:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575519.707810:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575519.707819:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569974144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575519.707831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569974016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575519.707835:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575519.707838:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575519.707839:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575519.707843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12273 00000100:00080000:3.0:1788575519.707845:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575519.707847:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575519.707850:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575519.707857:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569974272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575519.707867:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575519.707870:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575519.707871:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575519.707873:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12273 00000100:00080000:3.0:1788575519.707877:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575519.707879:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575519.707883:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575519.707890:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569974400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575519.707896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575519.707900:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575519.707907:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575519.707909:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575519.707912:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12273 00000100:00080000:3.0:1788575519.707915:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575519.707918:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575519.707921:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575519.707928:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569974528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575519.707937:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575519/real 1788575519] req@ffff920a529b2d80 x1875444569974016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575519.707955:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569974016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575519.707966:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575519.707970:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575519.707973:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569974016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575519.707996:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575519.708018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575519.708030:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575519/real 1788575519] req@ffff920a529b3480 x1875444569974144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575519.708042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569974144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575519.708050:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575519.708054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575519.708058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575519.708063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575519.708081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575519.708091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575519/real 1788575519] req@ffff920a529b2a00 x1875444569974272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575519.708103:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569974272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575519.708110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575519.708113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575519.708117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575519.708122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575519.708140:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575519.708150:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575519/real 1788575519] req@ffff920a529b1c00 x1875444569974400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575519.708160:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569974400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575519.708168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575519.708171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575519.708174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575519.708180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575519.708198:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575519.708208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575519/real 1788575519] req@ffff920a529b3100 x1875444569974528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575519.708218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569974528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575519.708226:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575519.708228:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575519.708231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575520.155753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575520.731662:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575521.179720:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575521.755675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575522.203742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575522.779713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575523.227755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575523.803703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575524.251765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575524.827741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575524.827749:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575524.827760:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575524.827765:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575524.827779:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12278 00000100:00080000:3.0:1788575524.827795:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575524.827805:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575524.827851:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569974656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575524.827880:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575524.827884:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575524.827888:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575524.827893:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12278 00000100:00080000:3.0:1788575524.827898:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575524.827902:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575524.827909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569974656:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575524.827915:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575524.827933:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569974784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575524.827957:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575524.827962:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575524.827964:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575524.827972:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12278 00000100:00080000:3.0:1788575524.827977:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575524.827982:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575524.827985:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575524.827994:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575524.828010:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569974912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575524.828020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575524/real 1788575524] req@ffff920a529b1f80 x1875444569974656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575524.828035:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569974656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575524.828043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575524.828045:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575524.828049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569974656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575524.828074:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974784:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575524.828090:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575524.828096:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:1.0:1788575524.828099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575524.828101:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575524.828108:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12278 00000100:00080000:3.0:1788575524.828113:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575524.828117:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575524/real 1788575524] req@ffff920a529b0000 x1875444569974784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575524.828120:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575524.828132:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569974784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575524.828138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575524.828140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575524.828142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575524.828146:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575524.828160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575524.828166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575524/real 1788575524] req@ffff920a529b0700 x1875444569974912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575524.828173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569974912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575524.828178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575524.828179:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575524.828181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569974912:192.168.202.157@tcp:38:kworker.0 00000008:00100000:3.0:1788575524.828277:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575524.828297:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569975040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575524.828366:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575524.828371:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575524.828374:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575524.828380:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12278 00000100:00080000:3.0:1788575524.828386:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575524.828391:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575524.828398:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575524.828403:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575524.828418:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569975168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575524.828480:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575524.828507:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575524/real 1788575524] req@ffff920a529b2300 x1875444569975040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575524.828524:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569975040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575524.828534:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575524.828539:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575524.828542:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575524.828564:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575524.828612:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575524.828634:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575524/real 1788575524] req@ffff920a529b0380 x1875444569975168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575524.828646:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569975168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575524.828653:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575524.828656:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575524.828660:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575525.275735:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575525.851700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575526.299719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575526.875748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575527.323765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575527.899657:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575528.347695:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575528.923708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575529.371745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575529.947682:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575529.947688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575529.947690:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575529.947694:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575529.947702:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12283 00000100:00080000:2.0:1788575529.947712:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575529.947718:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575529.947747:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569975296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575529.947767:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575529.947770:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575529.947772:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575529.947774:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12283 00000100:00080000:2.0:1788575529.947777:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575529.947780:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575529.947788:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575529.947797:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569975296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575529.947798:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569975424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575529.947808:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575529.947811:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575529.947812:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575529.947814:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12283 00000100:00080000:2.0:1788575529.947816:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575529.947818:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575529.947821:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575529.947829:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569975552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575529.947834:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575529.947836:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575529.947837:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575529.947840:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12283 00000100:00080000:2.0:1788575529.947842:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575529.947843:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575529.947846:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575529.947853:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569975680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575529.947859:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575529.947861:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575529.947862:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575529.947865:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575529.947868:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12283 00000100:00080000:2.0:1788575529.947871:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575529.947875:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575529.947880:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575529.947892:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569975808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575529.947899:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575529/real 1788575529] req@ffff920a60ac4000 x1875444569975296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575529.947916:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569975296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575529.947923:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575529.947926:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575529.947929:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569975296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575529.947945:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575529.947961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575529.947969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575529/real 1788575529] req@ffff920a60ac6a00 x1875444569975424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575529.947976:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569975424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575529.947980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575529.947982:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575529.947985:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575529.947988:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575529.947998:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575529.948002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575529/real 1788575529] req@ffff920a60ac5180 x1875444569975552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575529.948008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569975552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575529.948013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575529.948014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575529.948016:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575529.948019:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575529.948028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575529.948033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575529/real 1788575529] req@ffff920a60ac5500 x1875444569975680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575529.948039:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569975680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575529.948043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575529.948044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575529.948046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575529.948049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575529.948057:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575529.948063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575529/real 1788575529] req@ffff920a60ac7100 x1875444569975808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575529.948069:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569975808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575554 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575529.948073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575529.948075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575529.948076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569975808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575530.395724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575530.971682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575531.419779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575531.995678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575532.443763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575533.019743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575533.467771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575534.043713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575534.491712:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575535.067692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575535.067703:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.067710:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575535.067714:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.067724:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12288 00000100:00080000:0.0:1788575535.067734:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.067741:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575535.067778:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569975936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.067804:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.067808:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575535.067810:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.067814:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12288 00000100:00080000:0.0:1788575535.067818:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.067822:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575535.067831:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575535.067834:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569975936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575535.067846:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569976064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.067855:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.067859:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575535.067861:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.067865:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12288 00000100:00080000:0.0:1788575535.067868:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.067871:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575535.067875:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575535.067884:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569976192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.067891:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.067893:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575535.067894:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.067897:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12288 00000100:00080000:0.0:1788575535.067900:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.067904:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575535.067906:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575535.067909:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575535.067919:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569976320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575535.067926:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575535.067929:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575535.067932:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575535.067936:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12288 00000100:00080000:0.0:1788575535.067940:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575535.067944:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575535.067949:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575535.067959:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff920a48353480 x1875444569975936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575535.067962:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569976448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575535.068003:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569975936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.068015:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.068021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575535.068026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569975936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575535.068052:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.068076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575535.068087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff920a48351c00 x1875444569976064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.068097:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569976064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.068104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.068107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575535.068110:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575535.068115:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575535.068134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575535.068144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff920a48351500 x1875444569976192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.068154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569976192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.068160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.068162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575535.068165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575535.068171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575535.068193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575535.068206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff920a48350380 x1875444569976320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.068220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569976320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.068228:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.068232:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575535.068235:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575535.068240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575535.068263:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575535.068272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575535/real 1788575535] req@ffff920a48350a80 x1875444569976448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575535.068282:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569976448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575535.068289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575535.068291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575535.068294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575535.515765:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575536.091704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575536.539796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575537.115726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575537.563787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575538.139683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575538.587722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575539.163717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575539.611819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575540.187683:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575540.187683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575540.187711:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575540.187716:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.187729:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12293 00000100:00080000:0.0:1788575540.187740:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.187748:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575540.187793:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569976576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.187821:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.187825:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575540.187827:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.187831:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12293 00000100:00080000:0.0:1788575540.187835:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.187839:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575540.187851:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575540.187859:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569976576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575540.187865:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569976704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.187876:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.187880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575540.187882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.187893:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12293 00000100:00080000:0.0:1788575540.187897:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.187901:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575540.187905:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575540.187916:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569976832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.187925:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.187928:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575540.187929:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575540.187932:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12293 00000100:00080000:0.0:1788575540.187936:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.187940:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575540.187945:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575540.187958:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569976960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575540.187966:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575540.187970:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575540.187972:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575540.187977:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976704:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575540.187978:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12293 00000100:00080000:0.0:1788575540.187985:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575540.187990:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575540.187996:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575540.188008:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569977088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575540.188014:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575540.188041:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575540.188063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575540.635734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575541.211818:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575541.659851:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575542.235772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575542.683783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575543.259898:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:0.0:1788575543.260784:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:0.0:1788575543.260852:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1788575543.260871:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575543.260931:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575543.260943:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575543.260951:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:0.0:1788575543.260960:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:1.0:1788575543.261081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575543] req@ffff920a48350380 x1875444569977088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575543.261110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569977088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575543.261121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575543.261129:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575543.261133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977088:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575543.261140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575543] req@ffff920a48350700 x1875444569976704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575543.261153:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569976704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575543.261162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575543.261167:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575543.261170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575543.261180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575543] req@ffff920a48353800 x1875444569976832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575543.261198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569976832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575543.261206:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575543.261211:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575543.261216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575543.261227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575543] req@ffff920a48350a80 x1875444569976960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575543.261241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569976960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575543.261248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575543.261252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575543.261257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569976960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575543.261263:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575540/real 1788575543] req@ffff920a48353b80 x1875444569976576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575543.261274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569976576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575543.261280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575543.261283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575543.261286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569976576:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575543.707778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575544.283777:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575544.731740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575545.307731:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.307745:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575545.307750:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.307762:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12298 00000100:00080000:2.0:1788575545.307773:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.307793:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575545.307810:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788575545.307832:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569977216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.307859:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.307861:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575545.307863:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.307866:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12298 00000100:00080000:2.0:1788575545.307868:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.307871:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575545.307883:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575545.307896:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569977344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.307906:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.307909:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575545.307912:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575545.307915:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569977216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575545.307919:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12298 00000100:00080000:2.0:1788575545.307926:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.307931:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575545.307936:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575545.307950:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569977472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.307959:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.307962:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575545.307964:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.307967:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12298 00000100:00080000:2.0:1788575545.307971:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575545.307974:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575545.307980:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575545.307989:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569977600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575545.307998:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575545.308001:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575545.308003:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575545.308006:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12298 00000100:00080000:2.0:1788575545.308010:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575545.308011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575545.308017:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575545.308023:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575545.308033:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569977728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575545.308059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff920a60ac6300 x1875444569977216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.308080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569977216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.308091:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.308096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575545.308100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569977216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575545.308128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.308151:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575545.308165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff920a60ac4a80 x1875444569977344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.308175:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569977344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.308183:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.308186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575545.308190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575545.308195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575545.308213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575545.308223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff920a60ac7100 x1875444569977472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.308234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569977472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.308242:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.308245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575545.308248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575545.308254:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575545.308273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575545.308281:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff920a60ac5500 x1875444569977600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.308291:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569977600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.308299:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.308301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575545.308305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575545.308309:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575545.308326:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575545.308335:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575545/real 1788575545] req@ffff920a60ac5180 x1875444569977728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575545.308346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569977728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575545.308353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575545.308362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575545.308366:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575545.755733:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575546.331756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575546.779746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575547.355703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575547.803745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575548.379713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575548.827807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575549.403702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575549.851839:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575550.427720:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575550.427726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575550.427731:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575550.427736:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.427746:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12303 00000100:00080000:2.0:1788575550.427756:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.427764:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575550.427806:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569977856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.427835:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.427839:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575550.427841:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.427844:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12303 00000100:00080000:2.0:1788575550.427847:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.427850:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575550.427871:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575550.427882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569977984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.427890:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.427892:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575550.427894:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.427898:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12303 00000100:00080000:2.0:1788575550.427901:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.427903:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575550.427906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569977856:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575550.427907:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575550.427915:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569978112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.427921:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.427923:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575550.427924:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.427926:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12303 00000100:00080000:2.0:1788575550.427927:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.427929:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575550.427933:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575550.427939:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569978240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575550.427944:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575550.427946:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575550.427947:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575550.427949:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12303 00000100:00080000:2.0:1788575550.427951:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575550.427953:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575550.427956:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575550.427962:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569978368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575550.427989:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575550.428029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff920a60ac6a00 x1875444569977856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.428047:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569977856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.428058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.428063:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575550.428066:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569977856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575550.428092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575550.428115:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575550.428127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff920a60ac4000 x1875444569977984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.428138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569977984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.428147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.428150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575550.428153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569977984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575550.428159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575550.428176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575550.428186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff920a60ac4700 x1875444569978112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.428196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569978112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.428204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.428207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575550.428221:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575550.428227:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575550.428247:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575550.428258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff920a60ac4380 x1875444569978240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.428269:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569978240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.428277:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.428280:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575550.428283:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575550.428289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575550.428307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575550.428318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575550/real 1788575550] req@ffff920a60ac5c00 x1875444569978368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575550.428329:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569978368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575550.428337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575550.428340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575550.428343:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575550.875780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575551.451696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575551.899729:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575552.475743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575552.923784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575553.499740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575553.947778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575554.523690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575554.971788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575555.547685:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575555.547688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575555.547694:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575555.547698:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.547707:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12308 00000100:00080000:2.0:1788575555.547716:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.547722:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575555.547753:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569978496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.547775:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.547777:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575555.547779:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.547782:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12308 00000100:00080000:2.0:1788575555.547784:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.547787:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575555.547794:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575555.547802:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569978624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.547809:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.547813:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575555.547814:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.547817:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12308 00000100:00080000:2.0:1788575555.547818:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.547820:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575555.547824:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575555.547830:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569978752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575555.547841:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569978496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575555.547851:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.547856:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575555.547858:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.547865:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12308 00000100:00080000:2.0:1788575555.547868:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575555.547872:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575555.547878:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575555.547893:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569978880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575555.547903:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575555.547907:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575555.547909:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575555.547913:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12308 00000100:00080000:2.0:1788575555.547929:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1788575555.547944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575555.547949:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575555.547956:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575555.547971:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569979008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575555.547996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff920a60ac7800 x1875444569978496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.548013:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569978496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.548022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.548025:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575555.548028:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569978496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575555.548034:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575555.548052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575555.548060:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff920a60ac4e00 x1875444569978624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.548067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569978624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.548073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.548076:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575555.548079:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575555.548107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575555.548127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575555.548136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff920a60ac5c00 x1875444569978752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.548145:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569978752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.548151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.548153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575555.548155:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575555.548158:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575555.548170:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575555.548176:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff920a60ac4380 x1875444569978880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.548183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569978880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.548187:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.548189:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575555.548191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569978880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575555.548194:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575555.548203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575555.548208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575555/real 1788575555] req@ffff920a60ac4700 x1875444569979008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575555.548214:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569979008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575555.548222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575555.548224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575555.548226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575555.995797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575556.571714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575557.019767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575557.595683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575558.043756:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575558.619675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575559.067752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575559.643678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575560.091740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575560.667730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575560.667772:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575560.667780:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575560.667785:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575560.667799:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12314 00000100:00080000:0.0:1788575560.667810:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575560.667820:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575560.667860:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569979136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575560.667887:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575560.667890:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575560.667892:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575560.667895:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12314 00000100:00080000:0.0:1788575560.667899:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575560.667903:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575560.667914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575560.667927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569979136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575560.667929:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569979264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575560.667955:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575560.667958:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575560.667959:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575560.667964:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12314 00000100:00080000:0.0:1788575560.667967:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575560.667969:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575560.667972:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575560.667981:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569979392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575560.667990:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575560.667994:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575560.667996:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575560.668001:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12314 00000100:00080000:0.0:1788575560.668006:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575560.668011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575560.668014:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575560.668033:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575560.668058:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569979520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575560.668068:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575560/real 1788575560] req@ffff920a48351500 x1875444569979136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575560.668076:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575560.668088:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575560.668092:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575560.668101:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12314 00000100:00080000:0.0:1788575560.668107:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575560.668113:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575560.668119:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575560.668131:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569979648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575560.668131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569979136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575560.668164:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575560.668170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575560.668178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569979136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575560.668202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575560.668227:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575560.668239:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575560/real 1788575560] req@ffff920a48351c00 x1875444569979264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575560.668251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569979264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575560.668259:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575560.668261:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575560.668264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575560.668268:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575560.668283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575560.668295:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575560/real 1788575560] req@ffff920a48353480 x1875444569979392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575560.668304:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569979392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575560.668319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575560.668323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575560.668327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575560.668335:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575560.668374:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575560.668394:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575560/real 1788575560] req@ffff920a48353b80 x1875444569979520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575560.668407:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569979520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575560.668414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575560.668417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575560.668420:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575560.668425:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575560.668444:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575560.668455:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575560/real 1788575560] req@ffff920a48350a80 x1875444569979648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575560.668462:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569979648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575560.668468:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575560.668471:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575560.668475:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575561.115733:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575561.691654:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575562.139770:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575562.715722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575563.163778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575563.739717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575564.187789:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575564.763705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575565.211756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575565.787730:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575565.787741:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575565.787746:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575565.787758:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12319 00000100:00080000:0.0:1788575565.787774:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575565.787782:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575565.787785:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788575565.787836:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569979776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575565.787869:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575565.787873:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575565.787875:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575565.787880:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12319 00000100:00080000:0.0:1788575565.787884:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575565.787888:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575565.787900:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569979776:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575565.787901:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575565.787917:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569979904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575565.787934:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575565.787937:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575565.787939:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575565.787944:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12319 00000100:00080000:0.0:1788575565.787947:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575565.787952:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575565.787957:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575565.787968:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569980032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575565.787977:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575565.787980:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575565.787982:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575565.787984:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575565.787989:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12319 00000100:00080000:0.0:1788575565.787993:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575565.787998:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575565.788004:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575565.788018:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569980160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575565.788026:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575565/real 1788575565] req@ffff920a48353800 x1875444569979776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575565.788031:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575565.788041:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575565.788044:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575565.788049:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12319 00000100:00100000:2.0:1788575565.788052:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569979776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575565.788054:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575565.788062:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788575565.788065:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1788575565.788068:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575565.788070:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575565.788076:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569979776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575565.788084:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569980288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575565.788122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575565.788141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575565.788157:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575565/real 1788575565] req@ffff920a48350700 x1875444569979904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575565.788164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569979904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575565.788170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575565.788172:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575565.788174:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569979904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575565.788177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575565.788194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575565.788201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575565/real 1788575565] req@ffff920a48350380 x1875444569980032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575565.788208:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569980032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575565.788213:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575565.788214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575565.788216:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575565.788219:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575565.788230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575565.788234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575565/real 1788575565] req@ffff920a48350a80 x1875444569980160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575565.788240:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569980160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575565.788245:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575565.788246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575565.788248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575565.788251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575565.788260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575565.788265:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575565/real 1788575565] req@ffff920a48353b80 x1875444569980288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575565.788272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569980288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575565.788276:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575565.788278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575565.788280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575566.235796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575566.811697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575567.259744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575567.835703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575568.283788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575568.859730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575569.307791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575569.883709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575570.331775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575570.907707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575570.907746:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575570.907756:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575570.907762:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575570.907774:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12324 00000100:00080000:1.0:1788575570.907797:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575570.907807:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575570.907853:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444569980416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575570.907890:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575570.907893:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575570.907895:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575570.907900:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12324 00000100:00080000:1.0:1788575570.907904:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575570.907909:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575570.907920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575570.907932:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444569980544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575570.907941:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575570.907958:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575570.907959:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569980416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575570.907960:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575570.907969:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12324 00000100:00080000:1.0:1788575570.907973:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575570.907976:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575570.907981:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575570.907995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444569980672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575570.908006:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575570.908010:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575570.908012:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575570.908016:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12324 00000100:00080000:1.0:1788575570.908020:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575570.908024:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575570.908028:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575570.908038:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444569980800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575570.908044:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575570.908052:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575570.908056:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575570.908058:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575570.908063:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12324 00000100:00080000:1.0:1788575570.908067:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575570.908071:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575570.908076:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575570.908085:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444569980928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575570.908094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575570/real 1788575570] req@ffff920a52892a00 x1875444569980416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575570.908115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444569980416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575570.908126:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575570.908130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575570.908134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569980416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575570.908160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575570.908210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575570.908231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575570/real 1788575570] req@ffff920a52890a80 x1875444569980544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575570.908245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444569980544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575570.908252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575570.908254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575570.908257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575570.908261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575570.908278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575570.908289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575570/real 1788575570] req@ffff920a52892680 x1875444569980672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575570.908300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444569980672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575570.908309:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575570.908312:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575570.908315:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575570.908320:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575570.908336:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575570.908346:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575570/real 1788575570] req@ffff920a52890000 x1875444569980800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575570.908356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444569980800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575570.908364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575570.908368:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575570.908371:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575570.908377:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575570.908393:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575570.908400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575570/real 1788575570] req@ffff920a52893b80 x1875444569980928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575570.908408:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444569980928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575570.908413:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575570.908415:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575570.908416:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569980928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575571.355764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575571.931689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575572.379762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575572.955672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575573.403728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575573.979665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575574.427749:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575575.003695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575575.451738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575576.027687:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575576.027687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575576.027696:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575576.027700:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575576.027710:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12329 00000100:00080000:0.0:1788575576.027721:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575576.027728:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575576.027766:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569981056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575576.027791:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575576.027795:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575576.027797:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575576.027801:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12329 00000100:00080000:0.0:1788575576.027805:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575576.027809:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575576.027819:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575576.027831:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569981184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575576.027832:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569981056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575576.027841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575576.027845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575576.027846:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575576.027849:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12329 00000100:00080000:0.0:1788575576.027852:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575576.027854:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575576.027859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575576.027866:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569981312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575576.027874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575576.027877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575576.027879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575576.027882:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12329 00000100:00080000:0.0:1788575576.027884:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575576.027887:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575576.027891:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575576.027895:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575576.027900:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569981440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575576.027907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575576.027909:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575576.027911:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575576.027913:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12329 00000100:00080000:0.0:1788575576.027916:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575576.027919:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575576.027923:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575576.027931:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569981568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575576.027934:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff920a48353480 x1875444569981056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575576.027950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569981056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575576.027967:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575576.027975:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575576.027979:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569981056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575576.028001:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575576.028024:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575576.028034:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff920a48351c00 x1875444569981184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575576.028042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569981184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575576.028052:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575576.028054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575576.028057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575576.028063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575576.028081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575576.028091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff920a48351500 x1875444569981312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575576.028102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569981312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575576.028109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575576.028112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575576.028115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575576.028121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575576.028138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575576.028147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff920a48353b80 x1875444569981440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575576.028159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569981440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575576.028167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575576.028169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575576.028171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575576.028175:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575576.028188:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575576.028194:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575576/real 1788575576] req@ffff920a48350a80 x1875444569981568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575576.028201:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569981568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575576.028205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575576.028207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575576.028208:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575576.475725:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575577.051727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575577.499723:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575578.075736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575578.523775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575579.099668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575579.547724:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575580.123715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575580.571730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575581.147702:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575581.147703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575581.147713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575581.147717:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575581.147727:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12334 00000100:00080000:3.0:1788575581.147737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575581.147743:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575581.147780:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569981696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575581.147806:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575581.147810:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575581.147812:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575581.147816:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12334 00000100:00080000:3.0:1788575581.147825:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575581.147830:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575581.147840:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575581.147853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569981696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575581.147856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444569981824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575581.147872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575581.147876:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575581.147878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575581.147882:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12334 00000100:00080000:3.0:1788575581.147886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575581.147890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575581.147895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575581.147909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444569981952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575581.147913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575581.147925:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575581.147929:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575581.147931:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575581.147936:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12334 00000100:00080000:3.0:1788575581.147940:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575581.147944:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575581.147950:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575581.147951:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff920a529b3b80 x1875444569981696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575581.147966:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444569982080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575581.147968:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569981696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575581.147994:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575581.147995:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.148001:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:3.0:1788575581.148002:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575581.148006:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575581.148008:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569981696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575581.148013:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12334 00000100:00080000:3.0:1788575581.148019:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575581.148023:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575581.148028:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575581.148035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575581.148042:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444569982208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575581.148063:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575581.148076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff920a529b0e00 x1875444569981824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.148084:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444569981824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.148089:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.148091:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575581.148093:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575581.148097:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575581.148109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575581.148115:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff920a529b1500 x1875444569981952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.148125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444569981952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.148130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.148131:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575581.148133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569981952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575581.148136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575581.148146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575581.148151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff920a529b1180 x1875444569982080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.148157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444569982080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.148162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.148163:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575581.148165:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575581.148174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575581.148183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575581.148188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575581/real 1788575581] req@ffff920a529b3b80 x1875444569982208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575581.148194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444569982208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575581.148204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575581.148207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575581.148210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575581.595780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575582.171723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575582.619727:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575583.195690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575583.643754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575584.219669:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575584.667700:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575585.243697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575585.691740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575586.267668:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575586.267675:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575586.267679:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575586.267687:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12339 00000100:00080000:3.0:1788575586.267700:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575586.267707:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575586.267709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:3.0:1788575586.267737:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444569982336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575586.267756:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575586.267759:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575586.267760:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575586.267762:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12339 00000100:00080000:3.0:1788575586.267765:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575586.267767:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575586.267773:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575586.267783:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444569982464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575586.267793:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575586.267798:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:1.0:1788575586.267799:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569982336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575586.267800:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575586.267809:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12339 00000100:00080000:3.0:1788575586.267813:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575586.267817:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575586.267822:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575586.267834:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444569982592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575586.267844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575586.267848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575586.267850:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575586.267854:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12339 00000100:00080000:3.0:1788575586.267858:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575586.267861:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:1.0:1788575586.267866:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788575586.267869:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575586.267883:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444569982720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575586.267892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575586.267896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575586.267898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575586.267902:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff920a529b0380 x1875444569982336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575586.267904:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12339 00000100:00080000:3.0:1788575586.267910:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575586.267915:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575586.267921:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575586.267922:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444569982336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575586.267933:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444569982848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575586.267934:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.267943:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575586.267951:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569982336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575586.267974:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575586.267991:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575586.268001:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff920a529b2300 x1875444569982464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.268013:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444569982464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.268022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.268025:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575586.268029:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575586.268035:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575586.268051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575586.268058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff920a529b0700 x1875444569982592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.268065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444569982592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.268070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.268072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575586.268074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575586.268077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575586.268087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575586.268092:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff920a529b0000 x1875444569982720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.268098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444569982720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.268103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.268104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575586.268106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575586.268110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575586.268119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575586.268124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575586/real 1788575586] req@ffff920a529b1f80 x1875444569982848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575586.268130:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444569982848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575586.268135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575586.268137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575586.268138:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569982848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575586.715745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575587.291743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575587.739767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575588.315726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575588.763780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575589.339711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575589.787745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575590.363697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575590.811741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575591.387713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575591.387722:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575591.387727:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575591.387731:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575591.387738:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12344 00000100:00080000:1.0:1788575591.387747:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575591.387753:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575591.387781:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444569982976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575591.387809:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575591.387813:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575591.387815:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575591.387819:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12344 00000100:00080000:1.0:1788575591.387823:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575591.387828:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575591.387837:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575591.387850:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444569983104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575591.387852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569982976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575591.387866:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575591.387869:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575591.387870:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575591.387874:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12344 00000100:00080000:1.0:1788575591.387877:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575591.387880:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575591.387885:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575591.387898:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444569983232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575591.387907:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575591.387912:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788575591.387913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575591.387915:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575591.387921:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12344 00000100:00080000:1.0:1788575591.387926:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575591.387931:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575591.387937:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575591.387945:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575591] req@ffff920a52890380 x1875444569982976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575591.387951:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444569983360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575591.387964:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575591.387967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444569982976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575591.387969:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575591.387973:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575591.387980:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12344 00000100:00080000:1.0:1788575591.387986:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575591.387988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575591.387991:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788575591.387996:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:1.0:1788575591.387997:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575591.388002:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569982976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575591.388011:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444569983488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575591.388025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575591.388044:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575591.388053:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575591] req@ffff920a52891f80 x1875444569983104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575591.388061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444569983104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575591.388066:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575591.388068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575591.388070:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575591.388073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575591.388082:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575591.388087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575591] req@ffff920a52891880 x1875444569983232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575591.388094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444569983232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575591.388098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575591.388100:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575591.388101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575591.388104:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575591.388114:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575591.388119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575591] req@ffff920a52893480 x1875444569983360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575591.388125:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444569983360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575591.388137:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575591.388139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575591.388141:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575591.388144:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575591.388154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575591.388160:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575591/real 1788575591] req@ffff920a52891500 x1875444569983488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575591.388166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444569983488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575591.388170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575591.388171:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575591.388173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575591.835810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575592.411754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575592.859820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575593.435700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575593.883794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575594.459716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575594.907759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575595.483706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575595.931766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575596.507702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575596.507709:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575596.507716:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575596.507721:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575596.507730:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12349 00000100:00080000:1.0:1788575596.507740:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575596.507747:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575596.507781:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444569983616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575596.507816:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575596.507820:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575596.507822:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575596.507826:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12349 00000100:00080000:1.0:1788575596.507830:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575596.507835:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575596.507846:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575596.507861:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569983616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575596.507862:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444569983744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575596.507878:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575596.507882:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575596.507884:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575596.507888:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12349 00000100:00080000:1.0:1788575596.507892:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575596.507896:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575596.507902:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575596.507913:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569983872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575596.507923:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575596.507924:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575596.507928:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575596.507931:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575596.507935:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12349 00000100:00080000:1.0:1788575596.507939:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575596.507943:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575596.507948:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575596.507958:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569984000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575596.507967:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575596.507969:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575596.507971:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575596.507975:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12349 00000100:00080000:1.0:1788575596.507978:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575596.507982:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575596.507986:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575596.507997:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569984128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575596.508014:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff920a52890e00 x1875444569983616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575596.508040:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444569983616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575596.508049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575596.508052:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575596.508055:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569983616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575596.508072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575596.508185:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575596.508198:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff920a52893800 x1875444569983744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575596.508210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444569983744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575596.508220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575596.508264:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575596.508269:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575596.508277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575596.508297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575596.508308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff920a52892d80 x1875444569983872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575596.508319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569983872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575596.508331:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575596.508334:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575596.508337:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569983872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575596.508342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575596.508374:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575596.508385:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff920a52892300 x1875444569984000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575596.508396:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569984000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575596.508404:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575596.508408:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575596.508411:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575596.508416:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575596.508434:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575596.508444:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575596/real 1788575596] req@ffff920a52891180 x1875444569984128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575596.508454:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569984128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575596.508463:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575596.508465:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575596.508486:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575596.955712:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575597.531690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575597.979884:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575598.555682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575599.003745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575599.579727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575600.027772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575600.603693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575601.051755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575601.627698:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575601.627702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575601.627708:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575601.627713:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575601.627722:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12355 00000100:00080000:2.0:1788575601.627732:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575601.627739:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575601.627774:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569984256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575601.627805:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575601.627809:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575601.627811:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575601.627816:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12355 00000100:00080000:2.0:1788575601.627820:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575601.627824:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575601.627833:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575601.627847:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569984256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575601.627849:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569984384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575601.627862:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575601.627866:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575601.627868:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575601.627873:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12355 00000100:00080000:2.0:1788575601.627877:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575601.627881:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575601.627886:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575601.627898:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569984512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575601.627903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575601.627912:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575601.627915:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575601.627916:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575601.627920:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12355 00000100:00080000:2.0:1788575601.627924:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575601.627927:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575601.627932:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575601.627936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575601/real 1788575601] req@ffff920a60ac4000 x1875444569984256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575601.627943:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569984640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575601.627951:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575601.627953:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569984256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575601.627954:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575601.627957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575601.627963:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12355 00000100:00080000:2.0:1788575601.627967:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575601.627972:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575601.627975:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:3.0:1788575601.627976:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788575601.627982:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569984768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575601.627983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575601.627991:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569984256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575601.628015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575601.628032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575601.628040:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575601/real 1788575601] req@ffff920a60ac6a00 x1875444569984384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575601.628048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569984384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575601.628053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575601.628055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575601.628057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575601.628060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575601.628070:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575601.628076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575601/real 1788575601] req@ffff920a60ac5180 x1875444569984512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575601.628082:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569984512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575601.628087:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575601.628088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575601.628090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575601.628093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575601.628102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575601.628106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575601/real 1788575601] req@ffff920a60ac5500 x1875444569984640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575601.628113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569984640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575601.628117:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575601.628119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575601.628120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575601.628123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575601.628132:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575601.628137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575601/real 1788575601] req@ffff920a60ac7100 x1875444569984768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575601.628143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569984768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575601.628147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575601.628158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575601.628160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569984768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575602.075779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575602.651712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575603.099758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575603.675687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575604.123728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575604.699710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575605.147709:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575605.723670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575606.171704:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575606.747710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575606.747734:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575606.747742:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575606.747746:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575606.747756:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12360 00000100:00080000:2.0:1788575606.747765:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575606.747771:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575606.747806:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569984896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575606.747843:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575606.747846:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575606.747848:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575606.747851:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12360 00000100:00080000:2.0:1788575606.747854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575606.747858:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575606.747866:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575606.747880:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569985024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575606.747885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569984896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575606.747896:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575606.747900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575606.747902:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575606.747907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12360 00000100:00080000:2.0:1788575606.747910:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575606.747914:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575606.747919:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575606.747928:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569985152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575606.747934:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575606.747936:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575606.747937:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575606.747940:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12360 00000100:00080000:2.0:1788575606.747943:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575606.747945:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575606.747955:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575606.747971:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569985280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575606.747980:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575606.747984:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575606.747986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985024:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575606.747987:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575606.747997:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12360 00000100:00080000:2.0:1788575606.748001:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575606.748005:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575606.748010:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575606.748024:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569985408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575606.748026:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575606.748061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575606.748089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575607.195758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575607.771706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575608.219759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575608.795706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575609.243766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575609.819815:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788575609.819912:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788575609.819950:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575609.819965:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575609.820006:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575609.820016:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575609.820025:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575609.820032:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575609.820215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575606/real 1788575609] req@ffff920a60ac5180 x1875444569985408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.820240:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569985408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.820250:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.820255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575609.820259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985408:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575609.820266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575606/real 1788575609] req@ffff920a60ac6300 x1875444569985024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.820280:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569985024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.820415:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.820420:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575609.820424:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575609.820431:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575606/real 1788575609] req@ffff920a60ac7100 x1875444569985152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.820444:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569985152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.820451:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.820454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575609.820457:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575609.820463:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575606/real 1788575609] req@ffff920a60ac5500 x1875444569985280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.820472:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569985280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.820479:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.820481:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575609.820484:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575609.820489:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575606/real 1788575609] req@ffff920a60ac4a80 x1875444569984896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575609.820500:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569984896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575609.820538:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575609.820544:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575609.820546:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569984896:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788575610.267748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575610.843707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575611.291766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575611.867676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575611.867681:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575611.867688:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575611.867692:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575611.867703:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12365 00000100:00080000:2.0:1788575611.867713:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575611.867720:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575611.867758:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569985536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575611.867787:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575611.867790:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575611.867792:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575611.867796:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12365 00000100:00080000:2.0:1788575611.867799:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575611.867804:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575611.867813:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575611.867825:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569985664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575611.867835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569985536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575611.867847:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575611.867850:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575611.867852:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575611.867857:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12365 00000100:00080000:2.0:1788575611.867860:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575611.867864:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575611.867869:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575611.867880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569985792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575611.867889:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575611.867892:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575611.867894:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575611.867898:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575611.867902:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12365 00000100:00080000:2.0:1788575611.867907:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575611.867912:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575611.867917:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575611.867935:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575611/real 1788575611] req@ffff920a60ac4a80 x1875444569985536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575611.867936:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569985920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575611.867963:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575611.867966:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575611.867968:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575611.867970:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569985536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575611.867975:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12365 00000100:00080000:2.0:1788575611.867982:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575611.867987:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575611.867988:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788575611.867994:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:2.0:1788575611.867995:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575611.868001:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569985536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575611.868011:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569986048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575611.868026:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575611.868052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575611.868064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575611/real 1788575611] req@ffff920a60ac5500 x1875444569985664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575611.868075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569985664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575611.868082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575611.868085:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575611.868088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575611.868093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575611.868113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575611.868123:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575611/real 1788575611] req@ffff920a60ac7100 x1875444569985792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575611.868133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569985792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575611.868141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575611.868144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575611.868147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575611.868152:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575611.868171:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575611.868179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575611/real 1788575611] req@ffff920a60ac6300 x1875444569985920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575611.868190:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569985920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575611.868197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575611.868200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575611.868203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569985920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575611.868208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575611.868224:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575611.868233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575611/real 1788575611] req@ffff920a60ac5180 x1875444569986048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575611.868243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569986048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575611.868248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575611.868251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575611.868253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575612.315769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575612.891696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575613.339832:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575613.915671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575614.363736:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575614.939736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575615.387760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575615.963739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575616.412671:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575616.987704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575616.987722:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575616.987729:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575616.987733:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575616.987743:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12370 00000100:00080000:2.0:1788575616.987753:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575616.987759:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575616.987794:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569986176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575616.987818:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575616.987822:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575616.987823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575616.987828:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12370 00000100:00080000:2.0:1788575616.987832:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575616.987836:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575616.987844:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569986176:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575616.987847:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575616.987863:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569986304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575616.987874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575616.987877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575616.987879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575616.987883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12370 00000100:00080000:2.0:1788575616.987887:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575616.987891:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788575616.987896:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575616.987896:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575616.987910:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569986432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575616.987919:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575616.987922:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575616.987926:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575616/real 1788575616] req@ffff920a60ac6a00 x1875444569986176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575616.987926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575616.987936:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12370 00000100:00080000:2.0:1788575616.987941:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575616.987945:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569986176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575616.987949:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575616.987959:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575616.987959:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575616.987964:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575616.987968:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569986176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575616.987977:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569986560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575616.987986:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986304:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788575616.987988:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575616.987993:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575616.987995:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575616.988000:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12370 00000800:00000100:0.0:1788575616.988004:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575616.988006:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575616.988014:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575616.988015:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575616/real 1788575616] req@ffff920a60ac4000 x1875444569986304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575616.988025:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569986304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575616.988030:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1788575616.988031:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575616.988034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575616.988039:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575616.988046:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569986688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575616.988048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575616.988077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575616.988090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575616/real 1788575616] req@ffff920a60ac4700 x1875444569986432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575616.988098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569986432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575616.988103:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575616.988105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575616.988107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575616.988118:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575616.988137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575616.988147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575616/real 1788575616] req@ffff920a60ac4380 x1875444569986560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575616.988159:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569986560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575616.988166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575616.988169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575616.988172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575616.988177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575616.988195:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575616.988205:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575616/real 1788575616] req@ffff920a60ac5c00 x1875444569986688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575616.988223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569986688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575641 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575616.988230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575616.988233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575616.988236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575617.435878:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575618.011753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575618.459773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575619.035728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575619.483858:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575620.059713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575620.507760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575621.083805:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575621.531761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575622.107709:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575622.107712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575622.107722:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575622.107727:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575622.107740:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12375 00000100:00080000:2.0:1788575622.107751:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575622.107760:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575622.107806:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569986816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575622.107829:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575622.107832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575622.107833:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575622.107835:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12375 00000100:00080000:2.0:1788575622.107838:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575622.107841:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575622.107851:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575622.107863:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569986944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575622.107870:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569986816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575622.107885:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575622.107889:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575622.107891:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575622.107896:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12375 00000100:00080000:2.0:1788575622.107899:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575622.107902:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575622.107906:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575622.107917:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569987072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575622.107922:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575622.107924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575622.107926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575622.107928:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12375 00000100:00080000:2.0:1788575622.107936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575622.107939:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575622.107942:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575622.107950:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569987200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575622.107951:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575622.107959:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575622.107961:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575622.107963:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575622.107965:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12375 00000100:00080000:2.0:1788575622.107967:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575622.107970:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575622.107972:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575622.107978:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569987328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575622.108000:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff920a60ac4e00 x1875444569986816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.108018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569986816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575622.108027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575622.108031:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575622.108035:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569986816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575622.108059:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575622.108080:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575622.108091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff920a60ac7800 x1875444569986944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.108102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569986944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575622.108109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575622.108112:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575622.108115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569986944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575622.108120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575622.108138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575622.108147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff920a60ac5c00 x1875444569987072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.108158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569987072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575622.108165:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575622.108168:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575622.108171:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575622.108177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575622.108192:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575622.108199:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff920a60ac4380 x1875444569987200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.108205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569987200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575622.108214:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575622.108216:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575622.108218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575622.108222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575622.108239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575622.108248:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575622/real 1788575622] req@ffff920a60ac4700 x1875444569987328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575622.108259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569987328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575622.108266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575622.108269:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575622.108273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575622.555799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575623.131711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575623.579731:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575624.155744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575624.603750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575625.179693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575625.627758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575626.204293:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575626.651776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575627.227708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575627.227723:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.227730:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575627.227734:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.227745:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12380 00000100:00080000:0.0:1788575627.227754:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.227761:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575627.227806:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569987456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.227838:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.227841:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.227842:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.227846:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12380 00000100:00080000:0.0:1788575627.227848:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.227851:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575627.227859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.227868:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569987584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.227876:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.227880:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575627.227881:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.227887:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12380 00000100:00100000:3.0:1788575627.227888:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569987456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575627.227891:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.227899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575627.227904:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575627.227916:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569987712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575627.227925:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.227928:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575627.227929:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.227932:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12380 00000100:00080000:0.0:1788575627.227936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.227939:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575627.227944:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575627.227957:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569987840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575627.227963:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575627.227972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575627.227976:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575627.227977:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575627.227981:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12380 00000100:00080000:0.0:1788575627.227985:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575627.227988:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575627.227992:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575627.228001:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569987968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575627.228005:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff920a48350380 x1875444569987456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575627.228026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569987456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575627.228036:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575627.228040:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575627.228044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569987456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575627.228051:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575627.228072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575627.228083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff920a48350700 x1875444569987584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575627.228094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569987584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575627.228101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575627.228105:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575627.228108:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575627.228128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575627.228146:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575627.228156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff920a48353800 x1875444569987712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575627.228171:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569987712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575627.228179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575627.228182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575627.228184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575627.228190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575627.228206:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575627.228216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff920a48350a80 x1875444569987840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575627.228226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569987840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575627.228232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575627.228235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575627.228238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575627.228243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575627.228258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575627.228267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575627/real 1788575627] req@ffff920a48353b80 x1875444569987968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575627.228277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569987968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575627.228289:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575627.228292:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575627.228295:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569987968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575627.676793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575628.251730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575628.700180:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575629.275698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575629.723848:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575630.299709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575630.747787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575631.323693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575631.771740:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575632.347720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575632.347792:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.347797:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575632.347801:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.347810:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12385 00000100:00080000:0.0:1788575632.347821:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.347828:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575632.347861:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569988096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.347891:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.347895:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575632.347898:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.347903:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12385 00000100:00080000:0.0:1788575632.347908:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.347912:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575632.347922:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575632.347940:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569988096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575632.347942:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569988224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.347958:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.347962:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575632.347964:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.347968:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12385 00000100:00080000:0.0:1788575632.347972:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.347977:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575632.347983:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575632.347995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569988352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575632.348004:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.348007:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575632.348011:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575632.348012:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575632.348019:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12385 00000100:00080000:0.0:1788575632.348024:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575632.348029:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575632.348034:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575632.348048:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569988480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575632.348061:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff920a48351500 x1875444569988096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575632.348062:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575632.348070:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575632.348073:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575632.348082:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12385 00000100:00080000:0.0:1788575632.348087:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575632.348093:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569988096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575632.348096:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575632.348105:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575632.348109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575632.348115:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575632.348119:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569988608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575632.348122:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569988096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575632.348151:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575632.348175:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575632.348187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff920a48351c00 x1875444569988224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575632.348204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569988224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575632.348211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575632.348214:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575632.348217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575632.348222:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575632.348254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575632.348271:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff920a48353480 x1875444569988352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575632.348285:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569988352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575632.348294:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575632.348298:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575632.348301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575632.348307:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575632.348326:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575632.348336:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff920a48353b80 x1875444569988480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575632.348346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569988480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575632.348361:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575632.348365:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575632.348368:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575632.348372:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575632.348386:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575632.348393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575632/real 1788575632] req@ffff920a48350a80 x1875444569988608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575632.348406:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569988608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575632.348415:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575632.348419:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575632.348422:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575632.795856:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575633.371787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575633.819827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575634.395725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575634.843866:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575635.419684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575635.867795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575636.443713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575636.891849:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575637.467726:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575637.467727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575637.467738:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575637.467743:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575637.467756:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12390 00000100:00080000:2.0:1788575637.467767:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575637.467776:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575637.467815:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569988736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575637.467841:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575637.467844:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575637.467845:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575637.467848:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12390 00000100:00080000:2.0:1788575637.467850:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575637.467853:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575637.467862:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575637.467871:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569988864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575637.467882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575637.467886:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569988736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575637.467887:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575637.467891:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575637.467899:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12390 00000100:00080000:2.0:1788575637.467902:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575637.467906:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575637.467911:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575637.467923:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569988992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575637.467932:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575637.467935:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575637.467937:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575637.467941:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12390 00000100:00080000:2.0:1788575637.467955:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575637.467959:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575637.467963:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575637.467968:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575637.467975:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569989120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575637.467981:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575637.467984:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575637.467986:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575637.467989:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12390 00000100:00080000:2.0:1788575637.467995:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575637.467997:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575637.468000:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575637.468007:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569989248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575637.468011:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff920a60ac4000 x1875444569988736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575637.468030:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569988736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575637.468038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575637.468041:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575637.468044:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569988736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575637.468064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575637.468079:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575637.468087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff920a60ac6a00 x1875444569988864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575637.468094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569988864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575637.468099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575637.468101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575637.468103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575637.468107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575637.468116:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575637.468121:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff920a60ac5180 x1875444569988992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575637.468127:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569988992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575637.468132:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575637.468133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575637.468135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569988992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575637.468137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575637.468147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575637.468152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff920a60ac6300 x1875444569989120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575637.468158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569989120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575637.468163:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575637.468164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575637.468166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575637.468168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575637.468180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575637.468188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575637/real 1788575637] req@ffff920a60ac7100 x1875444569989248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575637.468199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569989248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575637.468207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575637.468210:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575637.468213:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575637.915811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575638.491720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575638.939741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575639.515762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575639.963940:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575640.539749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575640.987832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575641.563726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575642.011795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575642.587703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575642.587737:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575642.587742:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575642.587745:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575642.587753:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12395 00000100:00080000:0.0:1788575642.587761:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575642.587767:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575642.587797:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569989376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575642.587816:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575642.587819:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575642.587820:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575642.587822:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12395 00000100:00080000:0.0:1788575642.587824:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575642.587826:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575642.587834:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575642.587841:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569989504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575642.587848:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575642.587852:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575642.587853:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575642.587853:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569989376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575642.587860:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12395 00000100:00080000:0.0:1788575642.587863:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575642.587865:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575642.587867:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575642.587874:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569989632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575642.587879:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575642.587881:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575642.587882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575642.587885:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12395 00000100:00080000:0.0:1788575642.587887:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575642.587889:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575642.587891:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575642.587898:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569989760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575642.587902:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575642.587904:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575642.587906:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575642.587908:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12395 00000100:00080000:0.0:1788575642.587910:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575642.587912:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575642.587914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575642.587918:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575642.587922:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569989888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575642.587968:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff920a48353800 x1875444569989376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575642.587981:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569989376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575642.587988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575642.587990:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575642.587993:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569989376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575642.588011:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575642.588026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575642.588035:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff920a48350700 x1875444569989504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575642.588046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569989504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575642.588051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575642.588053:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575642.588055:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575642.588058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575642.588068:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575642.588073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff920a48350380 x1875444569989632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575642.588080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569989632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575642.588084:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575642.588086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575642.588087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575642.588090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575642.588100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575642.588105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff920a48350a80 x1875444569989760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575642.588110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569989760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575642.588119:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575642.588121:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575642.588123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575642.588125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575642.588134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575642.588138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575642/real 1788575642] req@ffff920a48353b80 x1875444569989888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575642.588144:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569989888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575642.588149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575642.588151:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575642.588152:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569989888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575643.035740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575643.611726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575644.059838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575644.635718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575645.083953:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575645.659732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575646.107803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575646.683722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575647.131811:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575647.707707:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575647.707708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575647.707717:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575647.707721:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575647.707732:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12401 00000100:00080000:0.0:1788575647.707742:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575647.707750:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575647.707790:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569990016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575647.707812:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575647.707814:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575647.707816:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575647.707818:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12401 00000100:00080000:0.0:1788575647.707820:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575647.707823:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575647.707832:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575647.707843:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569990144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575647.707843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569990016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575647.707853:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575647.707855:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575647.707856:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575647.707858:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12401 00000100:00080000:0.0:1788575647.707859:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575647.707862:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575647.707864:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575647.707870:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569990272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575647.707877:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575647.707879:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575647.707881:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575647.707882:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12401 00000100:00080000:0.0:1788575647.707885:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575647.707888:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575647.707891:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575647.707897:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569990400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575647.707902:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575647.707904:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575647.707905:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575647.707907:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12401 00000100:00080000:0.0:1788575647.707909:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1788575647.707912:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575647.707914:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575647.707918:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575647.707924:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569990528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575647.707954:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575647/real 1788575647] req@ffff920a48353480 x1875444569990016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575647.707974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569990016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575647.707985:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575647.707987:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575647.707990:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569990016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575647.708011:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575647.708028:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575647.708036:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575647/real 1788575647] req@ffff920a48351c00 x1875444569990144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575647.708044:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569990144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575647.708049:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575647.708052:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575647.708054:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575647.708057:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575647.708066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575647.708071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575647/real 1788575647] req@ffff920a48351500 x1875444569990272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575647.708078:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569990272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575647.708082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575647.708084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575647.708085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575647.708089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575647.708099:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575647.708104:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575647/real 1788575647] req@ffff920a48353b80 x1875444569990400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575647.708110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569990400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575647.708122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575647.708124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575647.708125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575647.708129:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575647.708139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575647.708145:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575647/real 1788575647] req@ffff920a48350a80 x1875444569990528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575647.708151:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569990528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575647.708156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575647.708157:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575647.708159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575648.155816:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575648.731739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575649.179804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575649.755753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575650.203802:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575650.779672:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575651.227817:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575651.803716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575652.251768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575652.827676:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575652.827676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575652.827693:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575652.827700:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575652.827709:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12406 00000100:00080000:0.0:1788575652.827719:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575652.827725:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575652.827761:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569990656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575652.827789:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575652.827793:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575652.827794:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575652.827799:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12406 00000100:00080000:0.0:1788575652.827803:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575652.827807:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575652.827817:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575652.827831:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569990784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575652.827839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569990656:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575652.827845:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575652.827850:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575652.827852:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575652.827857:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12406 00000100:00080000:0.0:1788575652.827862:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575652.827865:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575652.827871:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575652.827883:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569990912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575652.827892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575652.827896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575652.827899:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1788575652.827901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575652.827906:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12406 00000100:00080000:0.0:1788575652.827911:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575652.827916:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575652.827928:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575652.827941:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575652/real 1788575652] req@ffff920a48350380 x1875444569990656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575652.827947:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569991040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575652.827967:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575652.827972:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1788575652.827974:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569990656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575652.827975:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575652.827983:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12406 00000100:00080000:0.0:1788575652.827991:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575652.827994:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575652.827997:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:3.0:1788575652.828002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:0.0:1788575652.828004:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575652.828008:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569990656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575652.828020:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569991168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575652.828036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575652.828062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575652.828075:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575652/real 1788575652] req@ffff920a48350700 x1875444569990784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575652.828087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569990784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575652.828096:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575652.828099:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575652.828102:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575652.828107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575652.828126:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575652.828137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575652/real 1788575652] req@ffff920a48353800 x1875444569990912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575652.828148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569990912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575652.828162:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575652.828165:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575652.828169:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569990912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575652.828174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575652.828194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575652.828204:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575652/real 1788575652] req@ffff920a48350a80 x1875444569991040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575652.828216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569991040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575652.828223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575652.828226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575652.828229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575652.828234:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575652.828252:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575652.828262:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575652/real 1788575652] req@ffff920a48353b80 x1875444569991168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575652.828273:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569991168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575652.828280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575652.828283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575652.828287:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575653.275815:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575653.851704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575654.299791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575654.875699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575655.323794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575655.899712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575656.347780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575656.923708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575657.371803:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575657.947704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575657.947720:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575657.947731:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575657.947736:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575657.947748:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12411 00000100:00080000:0.0:1788575657.947760:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575657.947776:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575657.947833:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569991296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575657.947864:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575657.947868:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575657.947870:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575657.947873:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12411 00000100:00080000:0.0:1788575657.947876:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575657.947880:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575657.947890:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575657.947901:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569991424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575657.947909:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575657.947911:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575657.947913:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575657.947917:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569991296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575657.947918:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12411 00000100:00080000:0.0:1788575657.947923:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575657.947926:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575657.947929:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575657.947938:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569991552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575657.947953:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575657.947957:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575657.947958:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575657.947962:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12411 00000100:00080000:0.0:1788575657.947967:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575657.947971:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575657.947978:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575657.947991:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569991680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788575657.947999:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575657.948004:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575657.948016:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575657.948019:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575657.948024:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12411 00000100:00080000:0.0:1788575657.948028:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575657.948033:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575657.948038:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575657.948044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575657/real 1788575657] req@ffff920a48351500 x1875444569991296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575657.948055:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569991808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788575657.948066:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569991296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575657.948075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575657.948078:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575657.948083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569991296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575657.948111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575657.948134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575657.948146:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575657/real 1788575657] req@ffff920a48351c00 x1875444569991424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575657.948158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569991424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575657.948166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575657.948169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575657.948173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575657.948178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575657.948196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575657.948207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575657/real 1788575657] req@ffff920a48353480 x1875444569991552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575657.948217:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569991552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575657.948224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575657.948227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575657.948230:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575657.948235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575657.948254:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575657.948264:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575657/real 1788575657] req@ffff920a48353b80 x1875444569991680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575657.948274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569991680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575657.948282:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575657.948285:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575657.948288:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575657.948293:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575657.948310:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575657.948335:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575657/real 1788575657] req@ffff920a48350a80 x1875444569991808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575657.948347:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569991808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575682 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575657.948355:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575657.948358:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575657.948361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569991808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575658.396693:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575658.971759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575659.419808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575659.996419:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575660.443753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575661.020054:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575661.467818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575662.043734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575662.491737:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575663.067766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575663.067773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575663.067778:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575663.067786:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575663.067799:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12416 00000100:00080000:0.0:1788575663.067809:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575663.067817:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575663.067852:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569991936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575663.067882:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575663.067886:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575663.067888:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575663.067892:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12416 00000100:00080000:0.0:1788575663.067897:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575663.067901:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575663.067911:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575663.067925:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569992064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575663.067935:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575663.067939:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575663.067942:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575663.067943:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569991936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575663.067948:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12416 00000100:00080000:0.0:1788575663.067953:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575663.067957:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575663.067963:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575663.067975:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569992192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575663.067984:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575663.067988:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575663.067989:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575663.067993:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12416 00000100:00080000:0.0:1788575663.067997:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575663.068001:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575663.068006:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:3.0:1788575663.068011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575663.068022:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569992320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575663.068041:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575663.068044:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575663.068045:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1788575663.068048:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff920a48353800 x1875444569991936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575663.068050:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12416 00000100:00080000:0.0:1788575663.068056:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575663.068062:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575663.068067:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575663.068070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569991936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575663.068078:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569992448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575663.068086:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.068092:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575663.068096:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569991936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575663.068119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575663.068142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575663.068155:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff920a48350700 x1875444569992064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.068167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569992064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.068174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.068178:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575663.068180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575663.068187:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788575663.068207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575663.068218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff920a48350380 x1875444569992192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.068229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569992192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.068237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.068240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575663.068243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788575663.068258:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575663.068278:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575663.068287:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff920a48350a80 x1875444569992320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.068296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569992320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.068305:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.068307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575663.068310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788575663.068315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788575663.068332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575663.068341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575663/real 1788575663] req@ffff920a48353b80 x1875444569992448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575663.068351:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569992448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575663.068359:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788575663.068362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575663.068365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575663.515765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575664.091724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575664.539772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575665.115731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575665.563835:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575666.139702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575666.587803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575667.163726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575667.611844:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575668.187738:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575668.187742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575668.187753:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575668.187759:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.187772:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12421 00000100:00080000:0.0:1788575668.187785:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.187795:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575668.187857:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569992576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.187887:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.187891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575668.187893:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.187898:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12421 00000100:00080000:0.0:1788575668.187902:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.187906:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575668.187920:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575668.187932:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569992576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575668.187935:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569992704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.187949:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.187952:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575668.187955:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.187961:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12421 00000100:00080000:0.0:1788575668.187964:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.187969:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575668.187974:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575668.187986:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569992832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.187996:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.188000:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575668.188002:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.188006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12421 00000100:00080000:0.0:1788575668.188010:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.188015:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575668.188020:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575668.188030:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575668.188034:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569992960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575668.188045:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575668.188048:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575668.188051:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575668.188057:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12421 00000100:00080000:0.0:1788575668.188061:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575668.188066:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575668.188072:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575668.188077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff920a48353480 x1875444569992576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575668.188087:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569993088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575668.188104:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569992576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575668.188131:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575668.188137:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575668.188143:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569992576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575668.188171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575668.188196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575668.188212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff920a48351c00 x1875444569992704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575668.188226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569992704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575668.188235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575668.188238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575668.188241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575668.188248:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575668.188266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575668.188275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff920a48351500 x1875444569992832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575668.188286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569992832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575668.188293:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575668.188297:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575668.188299:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575668.188304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575668.188321:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575668.188331:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff920a48353b80 x1875444569992960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575668.188340:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569992960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575668.188352:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575668.188355:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575668.188357:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569992960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575668.188363:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575668.188379:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575668.188387:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575668/real 1788575668] req@ffff920a48350a80 x1875444569993088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575668.188397:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569993088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575668.188404:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575668.188407:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575668.188410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575668.635835:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575669.211814:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575669.659845:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575670.235768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575670.685517:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575671.259769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575671.707794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575672.283738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575672.731807:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575673.308260:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575673.308271:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575673.308636:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575673.308648:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12426 00000100:00080000:0.0:1788575673.308659:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575673.308669:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575673.308703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788575673.309060:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569993216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575673.309125:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569993216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575673.309240:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575673.309245:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575673.309247:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575673.309252:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12426 00000100:00080000:0.0:1788575673.309256:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575673.309261:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575673.309272:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575673.309288:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569993344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575673.309302:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575673.309305:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575673.309308:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575673.309313:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12426 00000100:00080000:0.0:1788575673.309316:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575673.309319:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575673.309325:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575673.309326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575673.309337:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569993472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575673.309348:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575673.309352:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575673.309354:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575673.309359:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12426 00000100:00080000:0.0:1788575673.309363:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575673.309368:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575673.309373:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575673.309375:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575673.309389:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569993600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575673.309399:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575673.309403:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575673.309405:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575673.309409:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993600:192.168.202.157@tcp:8:kworker.0 00000100:00080000:0.0:1788575673.309412:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12426 00000100:00080000:0.0:1788575673.309419:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575673.309424:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575673.309429:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575673.310048:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569993728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575673.310108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575673.755818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575674.331741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575674.779838:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575675.356283:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575675.804919:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575676.379740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575676.380093:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575676.380138:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575676.380155:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575676.380201:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575676.380212:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575676.380221:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575676.380230:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575676.380245:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575676] req@ffff920a48350a80 x1875444569993600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575676.380274:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569993600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575676.380287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575676.380293:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575676.380297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575676.380305:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575676] req@ffff920a48353800 x1875444569993472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575676.380316:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569993472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575676.380322:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575676.380326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575676.380329:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575676.380335:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575676] req@ffff920a48350700 x1875444569993344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575676.380346:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569993344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575676.380354:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575676.380358:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575676.380361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575676.380368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575676] req@ffff920a48350380 x1875444569993216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575676.380380:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569993216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575676.380388:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575676.380391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575676.380395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569993216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575676.380424:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575673/real 1788575676] req@ffff920a48353b80 x1875444569993728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575676.380436:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569993728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575676.380444:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575676.380462:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575676.380466:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575676.827791:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575677.403756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575677.851819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575678.427726:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.427735:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575678.427740:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.427750:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12431 00000100:00080000:2.0:1788575678.427760:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575678.427762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575678.427769:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575678.427804:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444569993856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.427834:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.427838:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575678.427840:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.427844:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12431 00000100:00080000:2.0:1788575678.427848:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.427852:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575678.427861:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575678.427873:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444569993984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.427882:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.427885:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575678.427887:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.427890:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12431 00000100:00080000:2.0:1788575678.427894:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.427898:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575678.427903:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575678.427914:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569994112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.427923:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.427926:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575678.427928:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.427933:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12431 00000100:00080000:2.0:1788575678.427937:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.427941:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575678.427947:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575678.427958:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569993856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575678.427960:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569994240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575678.427975:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575678.427978:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575678.427979:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575678.427983:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12431 00000100:00080000:2.0:1788575678.427987:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575678.427990:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575678.427995:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575678.428005:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569994368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575678.428026:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575678.428057:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff920a60ac5500 x1875444569993856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.428073:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444569993856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.428083:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.428087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575678.428091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569993856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575678.428098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575678.428117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575678.428127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff920a60ac4a80 x1875444569993984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.428143:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444569993984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.428151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.428154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575678.428156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569993984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575678.428162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575678.428177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575678.428186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff920a60ac7100 x1875444569994112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.428202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569994112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.428209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.428212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575678.428215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575678.428240:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575678.428258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575678.428267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff920a60ac6300 x1875444569994240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.428276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569994240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.428288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.428291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575678.428294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575678.428299:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575678.428313:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575678.428321:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575678/real 1788575678] req@ffff920a60ac5180 x1875444569994368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575678.428332:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569994368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575678.428339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575678.428342:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575678.428345:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575678.875759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575679.452085:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575679.899801:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575680.475961:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575680.923808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575681.499715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575681.947793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575682.524320:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575682.971969:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575683.547730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575683.547730:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575683.547741:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575683.547745:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575683.547756:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12436 00000100:00080000:2.0:1788575683.547766:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575683.547775:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575683.547812:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569994496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575683.547839:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575683.547843:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575683.547845:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575683.547849:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12436 00000100:00080000:2.0:1788575683.547852:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575683.547857:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575683.547866:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575683.547882:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569994624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575683.547891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569994496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575683.547895:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575683.547901:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575683.547902:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575683.547907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12436 00000100:00080000:2.0:1788575683.547911:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575683.547915:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575683.547920:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575683.547932:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569994752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575683.547941:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575683.547945:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575683.547947:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575683.547950:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12436 00000100:00080000:2.0:1788575683.547963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575683.547970:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788575683.547971:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:2.0:1788575683.547980:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575683.547993:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569994880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575683.548003:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575683.548006:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575683.548009:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575683.548011:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff920a60ac6a00 x1875444569994496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575683.548018:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12436 00000100:00080000:2.0:1788575683.548026:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575683.548032:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575683.548037:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575683.548038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569994496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.548051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1788575683.548052:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569995008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575683.548060:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575683.548069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569994496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575683.548096:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575683.548118:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575683.548130:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff920a60ac4000 x1875444569994624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.548142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569994624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.548149:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.548153:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575683.548156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575683.548161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575683.548179:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575683.548189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff920a60ac4700 x1875444569994752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.548199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569994752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.548207:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.548209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575683.548212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575683.548217:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575683.548236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575683.548246:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff920a60ac4380 x1875444569994880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.548258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569994880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.548265:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.548268:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575683.548271:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569994880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575683.548276:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575683.548293:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575683.548302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575683/real 1788575683] req@ffff920a60ac5c00 x1875444569995008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575683.548313:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569995008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575683.548320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575683.548323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575683.548326:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575683.995738:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575684.571707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575685.020345:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575685.595868:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575686.043787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575686.619687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575687.067775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575687.643693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575688.091743:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575688.667721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575688.667817:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575688.667823:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575688.667828:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575688.667837:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12442 00000100:00080000:2.0:1788575688.667847:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575688.667854:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575688.667887:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444569995136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575688.667918:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575688.667929:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575688.667931:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575688.667940:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12442 00000100:00080000:2.0:1788575688.667945:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575688.667950:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575688.667960:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575688.667973:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569995136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575688.667975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444569995264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575688.667989:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575688.667993:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575688.667995:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575688.667999:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12442 00000100:00080000:2.0:1788575688.668002:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575688.668006:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575688.668010:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575688.668023:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444569995392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575688.668032:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575688.668035:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1788575688.668039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575688.668040:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575688.668046:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12442 00000100:00080000:2.0:1788575688.668052:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575688.668056:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575688.668062:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575688.668070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575688/real 1788575688] req@ffff920a60ac7800 x1875444569995136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575688.668075:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444569995520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575688.668087:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444569995136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575688.668103:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575688.668105:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575688.668108:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575688.668110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:2.0:1788575688.668112:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575688.668115:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569995136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575688.668120:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12442 00000100:00080000:2.0:1788575688.668125:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575688.668129:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575688.668134:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575688.668137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575688.668150:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444569995648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575688.668158:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575688.668173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575688/real 1788575688] req@ffff920a60ac4e00 x1875444569995264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575688.668188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444569995264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575688.668196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575688.668199:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575688.668202:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575688.668208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575688.668226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575688.668234:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575688/real 1788575688] req@ffff920a60ac5c00 x1875444569995392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575688.668241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444569995392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575688.668246:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575688.668248:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575688.668249:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575688.668254:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575688.668271:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575688.668277:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575688/real 1788575688] req@ffff920a60ac4380 x1875444569995520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575688.668283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444569995520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575688.668287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575688.668289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575688.668290:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575688.668299:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575688.668309:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575688.668314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575688/real 1788575688] req@ffff920a60ac4700 x1875444569995648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575688.668320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444569995648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575688.668324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575688.668326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575688.668327:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575689.115799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575689.692006:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575690.140010:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575690.715751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575691.163812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575691.739690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575692.188697:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575692.763719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575693.211782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575693.787716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575693.787730:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575693.787737:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575693.787742:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575693.787753:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12447 00000100:00080000:1.0:1788575693.787763:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575693.787771:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575693.787808:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444569995776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575693.787833:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575693.787837:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575693.787839:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575693.787843:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12447 00000100:00080000:1.0:1788575693.787847:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575693.787851:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575693.787860:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575693.787872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569995776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575693.787875:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444569995904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575693.787887:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575693.787891:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575693.787893:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575693.787897:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12447 00000100:00080000:1.0:1788575693.787900:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575693.787904:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575693.787910:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575693.787922:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444569996032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575693.787931:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575693.787934:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575693.787936:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575693.787939:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12447 00000100:00080000:1.0:1788575693.787943:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575693.787945:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575693.787949:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575693.787957:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575693.787970:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444569996160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575693.787980:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575693.787983:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575693.787987:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575693.787988:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575693/real 1788575693] req@ffff920a52890700 x1875444569995776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575693.787993:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12447 00000100:00080000:1.0:1788575693.787999:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575693.788005:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575693.788011:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575693.788014:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444569995776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575693.788023:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444569996288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575693.788027:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575693.788035:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575693.788042:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569995776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575693.788065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575693.788089:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575693.788101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575693/real 1788575693] req@ffff920a52893100 x1875444569995904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575693.788113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444569995904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575693.788121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575693.788124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575693.788128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569995904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575693.788133:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575693.788148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575693.788156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575693/real 1788575693] req@ffff920a52891180 x1875444569996032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575693.788167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444569996032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575693.788174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575693.788177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575693.788180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575693.788185:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575693.788202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575693.788212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575693/real 1788575693] req@ffff920a52892300 x1875444569996160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575693.788223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444569996160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575693.788230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575693.788233:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575693.788236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575693.788242:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575693.788260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575693.788270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575693/real 1788575693] req@ffff920a52892d80 x1875444569996288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575693.788292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444569996288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575693.788306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575693.788310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575693.788313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575694.235767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575694.811681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575695.259781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575695.835708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575696.283791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575696.859715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575697.307711:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575697.883772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575698.331771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575698.907751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575698.907760:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575698.907768:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575698.907772:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575698.907782:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12452 00000100:00080000:3.0:1788575698.907792:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575698.907800:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575698.907837:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444569996416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575698.907863:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575698.907867:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575698.907870:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575698.907874:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12452 00000100:00080000:3.0:1788575698.907878:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575698.907882:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575698.907892:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575698.907906:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444569996544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575698.907910:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569996416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575698.907921:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575698.907924:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575698.907926:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575698.907931:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12452 00000100:00080000:3.0:1788575698.907935:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575698.907939:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575698.907944:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575698.907955:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569996672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575698.907964:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575698.907968:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575698.907970:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575698.907973:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12452 00000100:00080000:3.0:1788575698.907977:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575698.907981:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575698.907984:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575698.907991:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575698.908003:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569996800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575698.908011:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575698.908014:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575698.908016:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575698.908019:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575698/real 1788575698] req@ffff920a529b3100 x1875444569996416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575698.908022:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12452 00000100:00080000:3.0:1788575698.908029:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575698.908036:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575698.908041:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575698.908046:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444569996416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575698.908055:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569996928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575698.908059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575698.908068:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575698.908073:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569996416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575698.908098:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575698.908123:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575698.908137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575698/real 1788575698] req@ffff920a529b1c00 x1875444569996544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575698.908148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444569996544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575698.908156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575698.908161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575698.908164:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575698.908170:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575698.908190:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575698.908201:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575698/real 1788575698] req@ffff920a529b2a00 x1875444569996672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575698.908212:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569996672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575698.908220:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575698.908223:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575698.908226:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575698.908232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575698.908251:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575698.908261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575698/real 1788575698] req@ffff920a529b3480 x1875444569996800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575698.908272:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569996800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575698.908280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575698.908283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575698.908286:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575698.908290:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575698.908304:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575698.908314:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575698/real 1788575698] req@ffff920a529b2d80 x1875444569996928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575698.908324:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569996928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575698.908330:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575698.908333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575698.908336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569996928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575699.355772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575699.931705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575700.379849:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575700.955708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575701.403793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575701.979735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575702.427793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575703.003742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575703.451824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575704.027725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575704.027726:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575704.027737:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575704.027742:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575704.027754:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12457 00000100:00080000:0.0:1788575704.027767:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575704.027776:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575704.027817:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569997056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575704.027844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575704.027846:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575704.027848:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575704.027851:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12457 00000100:00080000:0.0:1788575704.027854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575704.027857:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575704.027867:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575704.027878:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569997184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575704.027886:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575704.027890:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575704.027892:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575704.027896:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12457 00000100:00080000:0.0:1788575704.027900:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575704.027905:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575704.027910:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575704.027917:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569997056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575704.027935:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569997312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575704.027947:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575704.027951:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575704.027953:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575704.027957:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12457 00000100:00080000:0.0:1788575704.027962:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575704.027967:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575704.027973:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575704.027985:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569997440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575704.027992:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575704.027995:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575704.027997:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575704.028001:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12457 00000100:00080000:0.0:1788575704.028004:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575704.028006:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575704.028009:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575704.028014:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575704.028021:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569997568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575704.028052:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff920a48351500 x1875444569997056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575704.028080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569997056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575704.028090:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575704.028095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575704.028098:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569997056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575704.028106:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575704.028127:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575704.028140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff920a48351c00 x1875444569997184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575704.028152:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569997184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575704.028159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575704.028162:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575704.028166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575704.028192:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575704.028211:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575704.028223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff920a48353480 x1875444569997312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575704.028235:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569997312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575704.028248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575704.028251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575704.028253:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575704.028257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575704.028273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575704.028280:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff920a48353b80 x1875444569997440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575704.028287:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569997440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575704.028292:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575704.028295:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575704.028297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575704.028300:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575704.028311:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575704.028317:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575704/real 1788575704] req@ffff920a48350380 x1875444569997568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575704.028323:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569997568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575704.028328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575704.028329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575704.028332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575704.475786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575705.051713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575705.499769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575706.075745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575706.523867:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575707.099744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575707.547820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575708.123731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575708.571869:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575709.147722:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575709.147734:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575709.147740:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575709.147752:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12462 00000100:00080000:0.0:1788575709.147764:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575709.147772:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575709.147818:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444569997696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575709.147825:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575709.147852:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575709.147856:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575709.147858:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575709.147863:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12462 00000100:00080000:0.0:1788575709.147866:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575709.147870:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575709.147880:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575709.147891:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444569997824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575709.147900:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575709.147904:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575709.147906:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575709.147910:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12462 00000100:00080000:0.0:1788575709.147922:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575709.147926:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575709.147930:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575709.147939:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444569997952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575709.147948:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575709.147951:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575709.147953:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575709.147956:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12462 00000100:00080000:0.0:1788575709.147959:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575709.147962:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575709.147966:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575709.147975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569998080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575709.147984:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575709.147987:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575709.147989:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575709.147992:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12462 00000100:00080000:0.0:1788575709.147995:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575709.147998:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575709.148002:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575709.148010:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569998208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575709.148485:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569997696:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575709.148561:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575709.148616:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff920a48350700 x1875444569997696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575709.148646:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444569997696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575709.148657:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575709.148661:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575709.148665:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569997696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575709.148672:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575709.148694:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575709.148707:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff920a48353800 x1875444569997824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575709.148719:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444569997824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575709.148727:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575709.148730:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575709.148733:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575709.148741:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575709.148762:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575709.148773:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff920a48350a80 x1875444569997952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575709.148784:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444569997952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575709.148791:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575709.148794:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575709.148797:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569997952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575709.148802:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575709.148822:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575709.148832:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff920a48350380 x1875444569998080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575709.148842:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569998080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575709.148849:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575709.148852:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575709.148855:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575709.148860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575709.148878:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575709.148889:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575709/real 1788575709] req@ffff920a48353b80 x1875444569998208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575709.148900:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569998208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575709.148907:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575709.148909:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575709.148912:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575709.595820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575710.171745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575710.619828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575711.195705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575711.643815:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575712.219701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575712.667848:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575713.243748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575713.691817:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575714.267736:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575714.267741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575714.267749:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575714.267754:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575714.267764:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12467 00000100:00080000:0.0:1788575714.267776:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.267783:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575714.267854:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444569998336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575714.267883:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575714.267886:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575714.267888:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575714.267891:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12467 00000100:00080000:0.0:1788575714.267895:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.267899:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575714.267909:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575714.267920:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569998336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575714.267922:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444569998464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575714.267935:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575714.267939:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575714.267940:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575714.267945:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12467 00000100:00080000:0.0:1788575714.267949:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.267953:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575714.267957:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575714.267970:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444569998592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575714.267979:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575714.267982:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575714.267983:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575714.267987:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12467 00000100:00080000:0.0:1788575714.267989:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.267992:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575714.267997:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575714.267997:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575714.268016:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444569998720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575714.268026:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575714.268029:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575714.268031:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575714.268035:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12467 00000100:00080000:0.0:1788575714.268039:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575714.268042:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575714.268047:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff920a48353480 x1875444569998336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575714.268048:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575714.268059:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444569998848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575714.268079:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444569998336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575714.268092:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575714.268096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575714.268099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569998336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575714.268122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575714.268140:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575714.268152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff920a48351c00 x1875444569998464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575714.268164:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444569998464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575714.268172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575714.268176:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575714.268179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575714.268184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575714.268204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575714.268215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff920a48351500 x1875444569998592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575714.268226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444569998592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575714.268233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575714.268236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575714.268239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575714.268244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575714.268261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575714.268269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff920a48353b80 x1875444569998720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575714.268275:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444569998720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575714.268280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575714.268282:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575714.268284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575714.268288:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575714.268297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575714.268302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575714/real 1788575714] req@ffff920a48350380 x1875444569998848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575714.268310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444569998848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575714.268317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575714.268319:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575714.268322:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569998848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575714.716215:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575715.291727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575715.739832:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575716.315739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575716.763823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575717.339730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575717.787792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575718.363759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575718.811879:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575719.387729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575719.387766:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.387774:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575719.387779:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.387791:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12472 00000100:00080000:2.0:1788575719.387803:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.387813:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575719.387862:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444569998976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.387892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.387896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575719.387898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.387902:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12472 00000100:00080000:2.0:1788575719.387904:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.387909:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575719.387921:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575719.387936:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444569999104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575719.387938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569998976:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575719.387954:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.387957:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575719.387958:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.387963:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12472 00000100:00080000:2.0:1788575719.387967:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.387971:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575719.387975:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575719.387988:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444569999232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.387996:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.387999:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575719.388001:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.388004:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12472 00000100:00080000:2.0:1788575719.388008:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.388012:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575719.388017:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575719.388021:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575719.388032:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444569999360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575719.388041:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575719.388044:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575719.388046:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575719.388050:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12472 00000100:00080000:2.0:1788575719.388054:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575719.388058:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575719.388063:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575719.388070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff920a60ac4000 x1875444569998976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575719.388074:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444569999488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575719.388102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444569998976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575719.388113:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575719.388117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575719.388119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569998976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575719.388143:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575719.388168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575719.388180:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff920a60ac6a00 x1875444569999104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575719.388191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444569999104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575719.388200:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575719.388203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575719.388206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575719.388211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575719.388226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575719.388236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff920a60ac5180 x1875444569999232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575719.388247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444569999232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575719.388255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575719.388258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575719.388262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575719.388267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575719.388287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575719.388297:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff920a60ac6300 x1875444569999360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575719.388308:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444569999360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575719.388316:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575719.388318:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575719.388321:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575719.388326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575719.388345:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575719.388354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575719/real 1788575719] req@ffff920a60ac7100 x1875444569999488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575719.388369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444569999488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575719.388376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575719.388379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575719.388395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575719.835787:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575720.412166:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575720.859757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575721.435732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575721.883871:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575722.459675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575722.907778:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575723.483731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575723.931858:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575724.507690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575724.507706:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575724.507713:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575724.507718:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575724.507730:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12477 00000100:00080000:3.0:1788575724.507742:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575724.507750:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575724.507807:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444569999616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575724.507836:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575724.507839:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575724.507841:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575724.507845:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12477 00000100:00080000:3.0:1788575724.507849:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575724.507854:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575724.507866:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575724.507876:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569999616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575724.507881:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444569999744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575724.507892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575724.507901:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575724.507903:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575724.507907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12477 00000100:00080000:3.0:1788575724.507911:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575724.507915:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575724.507919:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575724.507930:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444569999872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575724.507940:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575724.507944:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575724.507945:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575724.507951:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12477 00000800:00000100:0.0:1788575724.507952:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575724.507957:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575724.507963:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575724.507968:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575724.507978:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570000000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575724.507988:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575724.507991:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575724] req@ffff920a529b2d80 x1875444569999616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575724.507993:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575724.507997:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575724.508004:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12477 00000100:00080000:3.0:1788575724.508009:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788575724.508010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444569999616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575724.508015:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575724.508022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788575724.508023:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575724.508028:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575724.508032:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444569999616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575724.508037:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570000128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575724.508058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575724.508073:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575724.508081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575724] req@ffff920a529b3480 x1875444569999744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575724.508088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444569999744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575724.508093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575724.508095:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575724.508097:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575724.508099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575724.508109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575724.508114:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575724] req@ffff920a529b2a00 x1875444569999872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575724.508120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444569999872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575724.508125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575724.508126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575724.508128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444569999872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575724.508130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575724.508139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575724.508144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575724] req@ffff920a529b1c00 x1875444570000000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575724.508150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570000000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575724.508159:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575724.508161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575724.508163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575724.508165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575724.508174:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575724.508179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575724/real 1788575724] req@ffff920a529b3100 x1875444570000128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575724.508185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570000128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575724.508189:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575724.508191:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575724.508192:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575724.955782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575725.531723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575725.979793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575726.555721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575727.003725:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575727.579680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575728.027724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575728.603703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575729.051817:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575729.627700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575729.627715:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575729.627725:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575729.627731:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575729.627743:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12483 00000100:00080000:3.0:1788575729.627760:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575729.627768:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575729.627813:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570000256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575729.627841:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575729.627845:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575729.627847:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575729.627850:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12483 00000100:00080000:3.0:1788575729.627854:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575729.627858:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575729.627870:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575729.627881:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570000384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575729.627892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575729.627895:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575729.627898:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575729.627902:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570000256:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575729.627907:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12483 00000100:00080000:3.0:1788575729.627916:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575729.627921:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575729.627926:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575729.627939:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570000512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575729.627948:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575729.627951:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575729.627953:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575729.627957:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12483 00000100:00080000:3.0:1788575729.627961:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575729.627965:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575729.627983:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575729.627995:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570000640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575729.628005:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575729.628011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575729.628021:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575729.628024:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575729.628030:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12483 00000100:00080000:3.0:1788575729.628037:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575729.628042:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575729.628048:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575729.628064:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575729/real 1788575729] req@ffff920a529b1f80 x1875444570000256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575729.628065:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570000768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575729.628137:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570000256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575729.628150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575729.628154:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575729.628158:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570000256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575729.628180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575729.628202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575729.628215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575729/real 1788575729] req@ffff920a529b0000 x1875444570000384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575729.628225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570000384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575729.628233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575729.628236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575729.628239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575729.628244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575729.628261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575729.628272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575729/real 1788575729] req@ffff920a529b0700 x1875444570000512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575729.628296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570000512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575729.628304:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575729.628308:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575729.628310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575729.628317:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575729.628338:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575729.628348:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575729/real 1788575729] req@ffff920a529b2300 x1875444570000640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575729.628359:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570000640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575729.628366:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575729.628369:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575729.628372:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575729.628378:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575729.628397:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575729.628406:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575729/real 1788575729] req@ffff920a529b0380 x1875444570000768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575729.628426:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570000768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575729.628434:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575729.628437:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575729.628440:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570000768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575730.075738:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575730.651798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575731.099821:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575731.675714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575732.123884:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575732.699713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575733.147847:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575733.723726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575734.171795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575734.747730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575734.747734:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575734.747744:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575734.747750:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575734.747762:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12488 00000100:00080000:3.0:1788575734.747774:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575734.747785:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575734.747855:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570000896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575734.747877:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575734.747880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575734.747881:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575734.747884:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12488 00000100:00080000:3.0:1788575734.747887:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575734.747891:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575734.747901:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575734.747910:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570001024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575734.747918:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575734.747922:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570000896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575734.747927:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575734.747931:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575734.747935:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12488 00000100:00080000:3.0:1788575734.747939:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575734.747942:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575734.747947:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575734.747959:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570001152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575734.747969:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575734.747972:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575734.747974:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575734.747978:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12488 00000100:00080000:3.0:1788575734.747982:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575734.747986:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575734.747992:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575734.748001:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575734.748005:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570001280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575734.748017:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575734.748019:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575734.748021:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575734.748024:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12488 00000100:00080000:3.0:1788575734.748028:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575734.748032:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575734.748038:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575734.748044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575734/real 1788575734] req@ffff920a529b3b80 x1875444570000896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788575734.748053:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570001408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575734.748067:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570000896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575734.748079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575734.748083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575734.748086:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570000896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575734.748111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575734.748130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575734.748140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575734/real 1788575734] req@ffff920a529b1180 x1875444570001024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575734.748150:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570001024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575734.748157:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575734.748159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575734.748162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575734.748167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575734.748180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575734.748188:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575734/real 1788575734] req@ffff920a529b1500 x1875444570001152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575734.748197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570001152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575734.748203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575734.748205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575734.748208:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575734.748212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575734.748226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575734.748233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575734/real 1788575734] req@ffff920a529b0e00 x1875444570001280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575734.748242:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570001280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575734.748248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575734.748250:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575734.748252:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575734.748257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575734.748271:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575734.748278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575734/real 1788575734] req@ffff920a529b0380 x1875444570001408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575734.748286:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570001408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575734.748298:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575734.748301:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575734.748304:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575735.195876:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575735.771773:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575736.219812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575736.799018:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575737.243828:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575737.819711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575738.268227:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575738.843709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575739.291794:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575739.867685:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575739.867693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575739.867696:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575739.867702:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575739.867713:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12493 00000100:00080000:0.0:1788575739.867724:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575739.867731:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575739.867767:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570001536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575739.867793:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575739.867797:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575739.867799:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575739.867803:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12493 00000100:00080000:0.0:1788575739.867807:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575739.867812:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575739.867822:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575739.867827:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570001536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575739.867835:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570001664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575739.867843:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575739.867845:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575739.867848:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575739.867850:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12493 00000100:00080000:0.0:1788575739.867853:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575739.867856:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575739.867859:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575739.867866:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570001792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575739.867872:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575739.867874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575739.867876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575739.867878:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12493 00000100:00080000:0.0:1788575739.867880:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575739.867882:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575739.867885:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575739.867892:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570001920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575739.867898:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575739.867900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575739.867901:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575739.867903:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12493 00000100:00080000:0.0:1788575739.867905:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575739.867907:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575739.867910:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575739.867917:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570002048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575739.867929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575739.867966:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575739.867987:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575739.868013:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575740.315780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575740.891748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575741.339791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575741.915697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575742.363743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575742.939695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575742.939908:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575742.939952:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575742.939972:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575742.940020:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575742.940031:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575742.940039:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575742.940048:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575742.940116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575739/real 1788575742] req@ffff920a48353800 x1875444570001664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.940142:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570001664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.940156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.940161:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575742.940166:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575742.940173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575739/real 1788575742] req@ffff920a48350700 x1875444570001792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.940185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570001792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.940192:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.940196:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575742.940200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575742.940206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575739/real 1788575742] req@ffff920a48350380 x1875444570001920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.940218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570001920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.940224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.940227:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575742.940230:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570001920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575742.940236:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575739/real 1788575742] req@ffff920a48353b80 x1875444570002048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.940247:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570002048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.940255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.940258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575742.940261:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002048:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575742.940267:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575739/real 1788575742] req@ffff920a48350a80 x1875444570001536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575742.940279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570001536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575742.940286:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575742.940290:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575742.940294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570001536:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788575743.387720:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575743.963750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575744.411741:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575744.987691:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575744.987700:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575744.987705:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575744.987715:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12498 00000100:00080000:0.0:1788575744.987726:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575744.987729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575744.987735:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575744.987770:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570002176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575744.987794:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575744.987798:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575744.987800:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575744.987804:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12498 00000100:00080000:0.0:1788575744.987808:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575744.987813:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575744.987822:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575744.987827:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570002176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575744.987837:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570002304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575744.987847:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575744.987851:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575744.987853:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575744.987857:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12498 00000100:00080000:0.0:1788575744.987862:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575744.987867:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575744.987872:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575744.987882:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570002432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575744.987892:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575744.987904:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575744.987909:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575744.987912:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575744.987916:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12498 00000100:00080000:0.0:1788575744.987921:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575744.987925:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575744.987930:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575744.987936:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575744/real 1788575744] req@ffff920a48351500 x1875444570002176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575744.987945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570002560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575744.987959:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570002176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575744.987987:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575744.987988:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575744.987993:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575744.987997:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575744.987998:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575744.988005:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570002176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575744.988006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12498 00000100:00080000:0.0:1788575744.988013:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575744.988019:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575744.988025:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575744.988037:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575744.988038:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570002688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575744.988072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575744.988087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575744/real 1788575744] req@ffff920a48351c00 x1875444570002304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575744.988098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570002304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575744.988107:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575744.988111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575744.988114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575744.988120:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575744.988142:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575744.988152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575744/real 1788575744] req@ffff920a48353480 x1875444570002432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575744.988163:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570002432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575744.988170:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575744.988173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575744.988175:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575744.988181:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575744.988199:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575744.988209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575744/real 1788575744] req@ffff920a48350a80 x1875444570002560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575744.988234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570002560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575744.988242:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575744.988245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575744.988248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575744.988264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575744.988283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575744.988294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575744/real 1788575744] req@ffff920a48353b80 x1875444570002688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575744.988311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570002688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575769 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575744.988318:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575744.988321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575744.988325:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575745.435763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575746.011768:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575746.459792:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575747.035735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575747.483746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575748.059735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575748.507782:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575749.083675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575749.531719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575750.107695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575750.107711:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.107718:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575750.107722:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.107732:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12503 00000100:00080000:2.0:1788575750.107741:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.107748:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575750.107781:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570002816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.107804:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.107808:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575750.107810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.107814:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12503 00000100:00080000:2.0:1788575750.107818:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.107823:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575750.107834:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575750.107843:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570002816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575750.107850:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444570002944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.107872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.107875:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575750.107876:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.107880:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12503 00000100:00080000:2.0:1788575750.107883:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.107886:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575750.107891:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575750.107904:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570003072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575750.107913:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575750.107919:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.107924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575750.107926:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575750.107931:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12503 00000100:00080000:2.0:1788575750.107936:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575750.107940:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575750.107946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575750.107952:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff920a60ac4a80 x1875444570002816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575750.107960:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570003200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575750.107983:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575750.107987:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575750.107989:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575750.107994:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570002816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575750.107998:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12503 00000100:00080000:2.0:1788575750.108006:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575750.108010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575750.108013:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575750.108019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000008:00100000:2.0:1788575750.108021:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575750.108025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570002816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575750.108036:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570003328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575750.108049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575750.108075:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575750.108087:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff920a60ac5500 x1875444570002944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575750.108100:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444570002944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575750.108108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575750.108111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575750.108114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570002944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575750.108121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575750.108141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575750.108151:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff920a60ac7100 x1875444570003072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575750.108169:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570003072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575750.108178:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575750.108181:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575750.108184:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575750.108190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575750.108208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575750.108218:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff920a60ac6300 x1875444570003200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575750.108229:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570003200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575750.108237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575750.108240:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575750.108243:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575750.108257:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575750.108275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575750.108284:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575750/real 1788575750] req@ffff920a60ac5180 x1875444570003328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575750.108295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570003328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575750.108309:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575750.108312:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575750.108316:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575750.555779:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575751.131670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575751.579786:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575752.155702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575752.603771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575753.179759:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575753.627781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575754.203678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575754.651786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575755.227684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575755.227689:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.227696:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575755.227701:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.227711:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12508 00000100:00080000:2.0:1788575755.227732:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.227740:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575755.227780:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570003456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575755.227804:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.227807:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575755.227810:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.227814:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12508 00000100:00080000:2.0:1788575755.227818:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.227822:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575755.227831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570003456:192.168.202.157@tcp:250:kworker.0 00000008:00100000:2.0:1788575755.227832:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575755.227849:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570003584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575755.227867:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.227873:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575755.227876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.227880:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12508 00000100:00080000:2.0:1788575755.227884:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575755.227888:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575755.227888:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575755.227895:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575755.227905:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570003712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575755.227912:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575755.227915:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575755.227916:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575755.227919:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12508 00000100:00100000:0.0:1788575755.227923:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff920a60ac6a00 x1875444570003456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575755.227923:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.227928:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575755.227934:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575755.227948:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570003840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575755.227953:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570003456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575755.227968:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575755.227972:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575755.227973:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575755.227977:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575755.227978:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575755.227984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570003456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575755.227986:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12508 00000100:00080000:2.0:1788575755.227992:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575755.227996:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575755.228000:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575755.228006:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575755.228011:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570003968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575755.228029:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575755.228039:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff920a60ac4000 x1875444570003584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575755.228047:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570003584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575755.228053:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575755.228055:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575755.228057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575755.228061:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575755.228071:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575755.228077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff920a60ac4700 x1875444570003712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575755.228083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570003712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575755.228088:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575755.228090:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575755.228091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575755.228094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575755.228105:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575755.228110:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff920a60ac4380 x1875444570003840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575755.228116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570003840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575755.228121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575755.228123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575755.228124:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575755.228139:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575755.228153:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575755.228159:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575755/real 1788575755] req@ffff920a60ac5c00 x1875444570003968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575755.228170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570003968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575755.228175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575755.228177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575755.228179:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570003968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575755.675823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575756.251747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575756.699793:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575757.275683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575757.723772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575758.299680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575758.747806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575759.323665:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575759.771728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575760.347685:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575760.347721:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.347729:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575760.347734:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.347745:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12513 00000100:00080000:3.0:1788575760.347757:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.347766:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575760.347809:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570004096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575760.347843:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.347849:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575760.347851:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.347856:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12513 00000100:00080000:3.0:1788575760.347861:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.347866:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575760.347877:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575760.347891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570004096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575760.347899:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570004224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575760.347913:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.347917:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575760.347919:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.347924:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12513 00000100:00080000:3.0:1788575760.347929:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.347945:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575760.347952:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575760.347969:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570004352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575760.347975:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575760.347987:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.347991:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575760.347994:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575760.347999:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12513 00000100:00080000:3.0:1788575760.348005:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.348011:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575760.348013:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff920a529b2300 x1875444570004096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788575760.348021:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575760.348029:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570004096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575760.348038:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1788575760.348041:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570004480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575760.348043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575760.348049:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570004096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575760.348063:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575760.348067:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575760.348069:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575760.348073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004224:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575760.348076:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12513 00000100:00080000:3.0:1788575760.348082:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575760.348087:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575760.348092:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575760.348104:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788575760.348109:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570004608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575760.348116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff920a529b0700 x1875444570004224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575760.348136:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570004224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575760.348153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575760.348156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575760.348159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575760.348163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575760.348176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575760.348182:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff920a529b0000 x1875444570004352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575760.348195:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570004352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575760.348201:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575760.348203:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575760.348205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575760.348208:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575760.348220:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575760.348226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff920a529b1f80 x1875444570004480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575760.348232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570004480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575760.348236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575760.348238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575760.348240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575760.348249:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575760.348264:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575760.348270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575760/real 1788575760] req@ffff920a529b3100 x1875444570004608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575760.348276:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570004608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575760.348281:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575760.348283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575760.348284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575760.795760:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575761.371667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575761.819751:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575762.395680:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575762.843797:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575763.419649:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575763.867730:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575764.443705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575764.891695:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575765.467723:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575765.467732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575765.467737:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575765.467746:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575765.467758:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12518 00000100:00080000:1.0:1788575765.467770:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575765.467779:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575765.467829:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570004736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575765.467856:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575765.467860:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575765.467862:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575765.467867:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12518 00000100:00080000:1.0:1788575765.467870:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575765.467875:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575765.467887:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570004736:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788575765.467888:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575765.467904:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570004864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575765.467914:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575765.467917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575765.467919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575765.467924:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12518 00000100:00080000:1.0:1788575765.467927:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575765.467931:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575765.467936:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575765.467947:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570004992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575765.467956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575765.467966:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575765.467970:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575765.467972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575765.467977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12518 00000100:00080000:1.0:1788575765.467980:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575765.467984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575765.467990:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575765.467995:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff920a52893800 x1875444570004736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575765.468002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570005120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575765.468018:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575765.468021:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575765.468023:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575765.468026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570004736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575765.468029:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12518 00000100:00080000:1.0:1788575765.468034:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575765.468041:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575765.468047:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788575765.468048:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575765.468054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575765.468059:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570004736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575765.468062:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570005248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575765.468092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575765.468116:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575765.468128:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff920a52890e00 x1875444570004864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575765.468139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570004864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575765.468147:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575765.468150:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575765.468153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575765.468159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575765.468177:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575765.468187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff920a52891500 x1875444570004992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575765.468198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570004992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575765.468205:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575765.468209:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575765.468212:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570004992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575765.468232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575765.468250:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575765.468258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff920a52893480 x1875444570005120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575765.468265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570005120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575765.468270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575765.468272:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575765.468274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575765.468277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575765.468287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575765.468292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575765/real 1788575765] req@ffff920a52891880 x1875444570005248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575765.468298:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570005248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575765.468302:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575765.468304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575765.468305:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575765.915800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575766.491679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575766.939689:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575767.515692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575767.963759:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575768.539691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575768.987736:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575769.563684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575770.011771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575770.587683:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575770.587695:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575770.587700:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575770.587702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575770.587710:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12523 00000100:00080000:3.0:1788575770.587721:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575770.587729:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575770.587763:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570005376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575770.587786:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575770.587788:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575770.587790:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575770.587792:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12523 00000100:00080000:3.0:1788575770.587795:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575770.587798:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575770.587806:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575770.587814:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570005504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575770.587820:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575770.587822:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575770.587823:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575770.587825:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12523 00000100:00080000:3.0:1788575770.587826:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575770.587828:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575770.587832:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575770.587838:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570005632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575770.587844:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575770.587846:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575770.587847:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575770.587849:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12523 00000100:00080000:3.0:1788575770.587851:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575770.587856:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575770.587857:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570005376:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575770.587860:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575770.587877:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570005760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575770.587883:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575770.587885:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575770.587886:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575770.587888:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12523 00000100:00080000:3.0:1788575770.587890:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575770.587892:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575770.587895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575770.587903:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570005888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575770.587940:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575770.587992:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff920a529b1c00 x1875444570005376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575770.588010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570005376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575770.588020:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575770.588025:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575770.588028:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570005376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575770.588036:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575770.588058:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575770.588070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff920a529b2a00 x1875444570005504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575770.588096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570005504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575770.588104:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575770.588107:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575770.588111:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575770.588117:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575770.588137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575770.588148:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff920a529b3480 x1875444570005632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575770.588170:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570005632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575770.588179:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575770.588182:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575770.588185:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575770.588212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575770.588233:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575770.588245:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff920a529b2d80 x1875444570005760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575770.588258:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570005760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575770.588266:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575770.588270:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575770.588273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575770.588279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575770.588298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575770.588307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575770/real 1788575770] req@ffff920a529b3100 x1875444570005888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575770.588318:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570005888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575770.588325:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575770.588328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575770.588331:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570005888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575771.035762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575771.611677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575772.059719:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575772.635714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575773.083783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575773.659708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575774.107742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575774.683681:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575775.131785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575775.707836:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575775.707845:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575775.707849:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575775.707858:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12529 00000100:00080000:1.0:1788575775.707869:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575775.707876:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575775.707909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570006016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575775.707930:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575775.707934:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575775.707937:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575775.707941:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12529 00000100:00080000:1.0:1788575775.707944:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575775.707949:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575775.707958:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575775.707970:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570006144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575775.707978:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575775.707980:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575775.707982:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575775.707986:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12529 00000100:00080000:1.0:1788575775.707989:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575775.707991:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575775.707994:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575775.708000:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570006272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575775.708008:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575775.708011:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575775.708013:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575775.708016:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12529 00000100:00080000:1.0:1788575775.708019:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575775.708023:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575775.708028:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575775.708038:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570006400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575775.708047:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575775.708050:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575775.708052:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575775.708055:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12529 00000100:00080000:1.0:1788575775.708058:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575775.708062:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575775.708067:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575775.708079:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570006528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575775.708144:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788575775.708168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570006016:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788575775.708226:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575775.708256:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575775/real 1788575775] req@ffff920a52891f80 x1875444570006016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575775.708267:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570006016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575775.708275:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575775.708278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575775.708280:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570006016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575775.708284:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575775.708300:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575775.708308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575775/real 1788575775] req@ffff920a52890380 x1875444570006144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575775.708314:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570006144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575775.708319:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575775.708321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575775.708323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575775.708326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575775.708336:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575775.708342:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575775/real 1788575775] req@ffff920a52893b80 x1875444570006272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575775.708348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570006272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575775.708353:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575775.708354:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575775.708356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575775.708360:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575775.708371:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575775.708378:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575775/real 1788575775] req@ffff920a52890000 x1875444570006400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575775.708384:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570006400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575775.708388:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575775.708389:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575775.708391:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575775.708394:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575775.708402:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575775.708406:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575775/real 1788575775] req@ffff920a52892680 x1875444570006528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575775.708412:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570006528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575775.708415:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575775.708417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575775.708419:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575776.156788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575776.731689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575777.179739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575777.755700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575778.203757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575778.779663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575779.227739:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575779.803675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575780.251770:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575780.827702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575780.828021:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575780.828027:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575780.828031:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575780.828040:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12534 00000100:00080000:3.0:1788575780.828050:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575780.828058:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575780.828091:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570006656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575780.828118:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575780.828122:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575780.828124:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575780.828128:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12534 00000100:00080000:3.0:1788575780.828132:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575780.828136:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575780.828146:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575780.828160:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570006656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575780.828164:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570006784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575780.828177:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575780.828182:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575780.828184:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575780.828188:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12534 00000100:00080000:3.0:1788575780.828191:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575780.828195:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575780.828201:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575780.828210:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570006912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575780.828224:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1788575780.828225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575780.828231:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575780.828234:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575780.828239:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12534 00000100:00080000:3.0:1788575780.828244:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575780.828249:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575780.828254:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575780.828270:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570007040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575780.828271:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575780/real 1788575780] req@ffff920a529b1f80 x1875444570006656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575780.828294:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575780.828299:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575780.828301:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575780.828305:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12534 00000100:00100000:0.0:1788575780.828309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570006656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575780.828311:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575780.828320:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:0.0:1788575780.828328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1788575780.828329:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575780.828337:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575780.828343:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570006656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788575780.828346:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570007168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575780.828370:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575780.828401:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575780.828412:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575780/real 1788575780] req@ffff920a529b0000 x1875444570006784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575780.828424:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570006784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575780.828432:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575780.828435:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575780.828438:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575780.828444:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575780.828461:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575780.828497:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575780/real 1788575780] req@ffff920a529b0700 x1875444570006912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575780.828515:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570006912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575780.828522:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575780.828525:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575780.828528:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570006912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575780.828533:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575780.828551:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575780.828560:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575780/real 1788575780] req@ffff920a529b2300 x1875444570007040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575780.828572:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570007040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575780.828579:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575780.828582:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575780.828600:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575780.828604:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575780.828619:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575780.828626:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575780/real 1788575780] req@ffff920a529b0380 x1875444570007168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575780.828633:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570007168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575780.828642:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575780.828644:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575780.828646:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575781.276839:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575781.852515:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575782.300415:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575782.875720:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575783.323758:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575783.899751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575784.347749:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575784.923718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575785.371771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575785.947716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575785.947742:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575785.947749:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575785.947753:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575785.947763:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12539 00000100:00080000:1.0:1788575785.947773:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575785.947781:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575785.947815:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444570007296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575785.947851:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575785.947854:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575785.947856:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575785.947859:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12539 00000100:00080000:1.0:1788575785.947862:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575785.947865:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575785.947874:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575785.947886:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444570007424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575785.947893:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575785.947897:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575785.947898:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575785.947902:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12539 00000100:00080000:1.0:1788575785.947904:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575785.947905:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575785.947908:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575785.947915:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570007552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575785.947921:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575785.947924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575785.947925:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575785.947934:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12539 00000100:00100000:0.0:1788575785.947935:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570007296:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575785.947938:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575785.947945:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575785.947955:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575785.947962:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570007680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575785.947967:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575785.947970:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575785.947971:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575785.947973:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12539 00000100:00080000:1.0:1788575785.947975:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575785.947977:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575785.947980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575785.947986:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570007808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575785.948046:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575785.948102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575785/real 1788575785] req@ffff920a52890a80 x1875444570007296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575785.948124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444570007296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575785.948135:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575785.948140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575785.948144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570007296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575785.948153:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575785.948192:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575785.948204:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575785/real 1788575785] req@ffff920a52892a00 x1875444570007424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575785.948216:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444570007424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575785.948224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575785.948229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575785.948231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575785.948261:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575785.948282:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575785.948293:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575785/real 1788575785] req@ffff920a52892680 x1875444570007552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575785.948304:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570007552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575785.948312:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575785.948336:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575785.948339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575785.948346:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575785.948367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575785.948379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575785/real 1788575785] req@ffff920a52890000 x1875444570007680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575785.948390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570007680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575785.948398:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575785.948401:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575785.948404:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575785.948410:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575785.948430:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575785.948441:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575785/real 1788575785] req@ffff920a52893b80 x1875444570007808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575785.948453:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570007808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575810 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575785.948461:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575785.948464:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575785.948467:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570007808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575786.395740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575786.971718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575787.419810:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575787.995734:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575788.443764:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575789.019715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575789.467748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575790.043702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575790.491771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575791.067735:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575791.067735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575791.067749:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575791.067755:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.067768:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12544 00000100:00080000:2.0:1788575791.067779:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.067787:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575791.067833:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444570007936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.067862:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.067866:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.067868:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.067872:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12544 00000100:00080000:2.0:1788575791.067876:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.067880:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575791.067891:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.067906:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444570008064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.067917:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.067921:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575791.067923:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575791.067929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570007936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575791.067932:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12544 00000100:00080000:2.0:1788575791.067939:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.067944:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575791.067948:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575791.067966:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570008192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.067972:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.067975:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575791.067976:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.067979:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12544 00000100:00080000:2.0:1788575791.067981:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.067984:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575791.067987:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575791.067993:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570008320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575791.067999:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575791.068002:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575791.068003:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575791.068006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12544 00000100:00080000:2.0:1788575791.068008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575791.068012:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575791.068018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575791.068020:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575791.068027:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570008448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575791.068070:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575791] req@ffff920a60ac4e00 x1875444570007936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575791.068088:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444570007936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575791.068099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575791.068103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575791.068107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570007936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575791.068136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575791.068160:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575791.068169:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575791] req@ffff920a60ac7800 x1875444570008064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575791.068196:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444570008064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575791.068202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575791.068204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575791.068206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575791.068212:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575791.068232:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575791.068244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575791] req@ffff920a60ac5c00 x1875444570008192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575791.068255:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570008192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575791.068261:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575791.068263:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575791.068265:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575791.068269:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575791.068283:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575791.068289:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575791] req@ffff920a60ac4380 x1875444570008320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575791.068295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570008320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575791.068304:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575791.068306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575791.068308:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575791.068311:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575791.068320:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575791.068325:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575791/real 1788575791] req@ffff920a60ac4700 x1875444570008448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575791.068331:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570008448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575791.068335:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575791.068337:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575791.068339:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575791.515778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575792.091736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575792.539773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575793.115691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575793.563755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575794.139683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575794.587740:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575795.163690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575795.611786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575796.187686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575796.187688:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575796.187695:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575796.187699:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575796.187707:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12549 00000100:00080000:3.0:1788575796.187716:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575796.187722:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575796.187748:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570008576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575796.187769:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575796.187771:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575796.187773:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575796.187775:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12549 00000100:00080000:3.0:1788575796.187777:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575796.187780:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575796.187786:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575796.187793:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570008704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575796.187799:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575796.187802:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575796.187803:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575796.187806:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12549 00000100:00080000:3.0:1788575796.187807:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575796.187809:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575796.187811:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575796.187817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570008832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575796.187823:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575796.187828:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1788575796.187830:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570008576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575796.187830:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575796.187836:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12549 00000100:00080000:3.0:1788575796.187840:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575796.187842:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575796.187845:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575796.187853:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570008960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575796.187858:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575796.187861:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575796.187862:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575796.187864:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12549 00000100:00080000:3.0:1788575796.187866:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575796.187868:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575796.187870:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575796.187885:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570009088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575796.187894:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575796.187925:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff920a529b0e00 x1875444570008576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.187942:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570008576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.187952:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.187956:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575796.187959:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570008576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575796.187967:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575796.187986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575796.187997:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff920a529b1500 x1875444570008704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.188009:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570008704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.188018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.188021:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575796.188024:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575796.188063:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575796.188081:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575796.188090:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff920a529b1180 x1875444570008832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.188099:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570008832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.188107:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.188110:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575796.188113:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575796.188119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575796.188138:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575796.188149:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff920a529b3b80 x1875444570008960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.188161:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570008960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.188167:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.188170:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575796.188173:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570008960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575796.188178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575796.188197:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575796.188207:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575796/real 1788575796] req@ffff920a529b0380 x1875444570009088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575796.188218:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570009088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575796.188235:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575796.188239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575796.188242:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575796.635766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575797.211862:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575797.659819:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575798.235749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575798.683748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575799.259713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575799.707754:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575800.283683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575800.731776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575801.307697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575801.307699:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.307706:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575801.307710:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.307718:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12554 00000100:00080000:0.0:1788575801.307726:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.307731:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575801.307760:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570009216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.307784:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.307787:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575801.307788:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.307791:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12554 00000100:00080000:0.0:1788575801.307793:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.307795:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575801.307802:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575801.307809:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570009344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.307816:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.307818:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575801.307820:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.307822:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12554 00000100:00080000:0.0:1788575801.307824:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.307828:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575801.307829:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570009216:192.168.202.157@tcp:250:kworker.0 00000008:00100000:0.0:1788575801.307831:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575801.307840:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570009472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.307846:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.307848:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575801.307850:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.307852:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12554 00000100:00080000:0.0:1788575801.307855:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575801.307857:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575801.307861:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575801.307870:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570009600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575801.307875:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575801.307877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575801.307878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575801.307881:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12554 00000100:00080000:0.0:1788575801.307884:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575801.307884:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575801.307889:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575801.307892:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575801.307910:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570009728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575801.307927:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff920a48350380 x1875444570009216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575801.307941:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570009216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575801.307948:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575801.307951:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575801.307954:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570009216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575801.307963:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575801.307985:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575801.307997:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff920a48350700 x1875444570009344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575801.308010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570009344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575801.308017:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575801.308019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575801.308021:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575801.308039:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575801.308052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575801.308059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff920a48353800 x1875444570009472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575801.308065:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570009472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575801.308070:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575801.308072:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575801.308074:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575801.308077:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575801.308086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575801.308091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff920a48353b80 x1875444570009600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575801.308098:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570009600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575801.308102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575801.308103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575801.308105:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575801.308108:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575801.308116:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575801.308120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575801/real 1788575801] req@ffff920a48350a80 x1875444570009728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575801.308126:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570009728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575801.308130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575801.308132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575801.308134:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575801.755743:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575802.331688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575802.779775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575803.355692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575803.803768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575804.379699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575804.827765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575805.403687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575805.851712:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575806.427659:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575806.427667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575806.427668:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575806.427672:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575806.427682:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12559 00000100:00080000:0.0:1788575806.427691:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575806.427706:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575806.427739:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570009856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575806.427781:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575806.427785:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575806.427786:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575806.427789:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12559 00000100:00080000:0.0:1788575806.427792:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575806.427794:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575806.427802:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575806.427812:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570009984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575806.427822:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575806.427823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570009856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575806.427829:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575806.427833:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575806.427840:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12559 00000100:00080000:0.0:1788575806.427843:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575806.427847:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575806.427852:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575806.427865:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570010112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575806.427874:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575806.427877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575806.427878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575806.427882:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12559 00000100:00080000:0.0:1788575806.427885:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575806.427887:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575806.427893:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575806.427902:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570010240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575806.427907:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575806.427910:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575806.427910:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575806.427912:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12559 00000100:00080000:0.0:1788575806.427914:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575806.427916:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575806.427918:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575806.427924:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570010368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575806.427941:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575806.427977:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575806.427997:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575806.428017:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575806.875793:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575807.451663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575807.899766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575808.475813:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575808.923783:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575809.499725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788575809.499866:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788575809.499904:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788575809.499922:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575809.499959:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575809.499968:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575809.499979:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788575809.499989:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575809.499997:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575809] req@ffff920a48351c00 x1875444570009984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575809.500021:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570009984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575809.500033:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575809.500038:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575809.500042:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570009984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575809.500050:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575809] req@ffff920a48351500 x1875444570010112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575809.500061:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570010112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575809.500069:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575809.500073:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575809.500077:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575809.500083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575809] req@ffff920a48350a80 x1875444570010240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575809.500094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570010240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575809.500101:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575809.500104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575809.500107:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575809.500113:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575809] req@ffff920a48353b80 x1875444570010368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575809.500123:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570010368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575809.500130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575809.500133:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575809.500136:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010368:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575809.500143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575806/real 1788575809] req@ffff920a48353480 x1875444570009856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575809.500154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570009856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575809.500161:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575809.500164:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575809.500180:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570009856:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788575809.947827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575810.523707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575810.971781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575811.547693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575811.547735:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575811.547741:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575811.547746:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575811.547755:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12564 00000100:00080000:3.0:1788575811.547765:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575811.547774:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575811.547805:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570010496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575811.547839:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575811.547843:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575811.547845:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575811.547851:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12564 00000100:00080000:3.0:1788575811.547856:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575811.547861:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575811.547870:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575811.547883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570010624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575811.547892:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575811.547896:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575811.547899:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575811.547903:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12564 00000100:00080000:3.0:1788575811.547907:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575811.547911:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575811.547917:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575811.547929:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570010752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575811.547938:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575811.547941:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575811.547943:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575811.547946:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12564 00000100:00080000:3.0:1788575811.547949:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575811.547953:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575811.547958:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575811.547967:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570010880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575811.547975:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575811.547978:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575811.547980:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575811.547983:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12564 00000100:00080000:3.0:1788575811.547986:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575811.547989:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575811.547993:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575811.548002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570011008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575811.548073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570010496:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788575811.548137:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575811.548165:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff920a529b2300 x1875444570010496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575811.548180:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570010496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575811.548190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575811.548193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575811.548197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570010496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575811.548203:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575811.548222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575811.548233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff920a529b0700 x1875444570010624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575811.548244:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570010624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575811.548251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575811.548254:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575811.548257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575811.548262:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575811.548277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575811.548286:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff920a529b0000 x1875444570010752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575811.548295:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570010752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575811.548303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575811.548306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575811.548309:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575811.548313:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575811.548329:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575811.548337:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff920a529b1f80 x1875444570010880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575811.548353:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570010880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575811.548360:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575811.548363:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575811.548365:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570010880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575811.548370:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575811.548385:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575811.548393:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575811/real 1788575811] req@ffff920a529b3100 x1875444570011008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575811.548403:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570011008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575811.548410:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575811.548413:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575811.548415:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575811.998674:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575812.571708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575813.019842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575813.595713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575814.043745:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575814.619731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575815.067773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575815.644631:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575816.091820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575816.667683:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575816.667697:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575816.667703:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575816.667706:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575816.667717:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12570 00000100:00080000:0.0:1788575816.667726:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575816.667733:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575816.667763:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570011136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575816.667784:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575816.667788:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575816.667790:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575816.667794:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12570 00000100:00080000:0.0:1788575816.667797:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575816.667801:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575816.667810:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575816.667815:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570011136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575816.667822:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570011264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575816.667830:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575816.667832:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575816.667834:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575816.667837:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12570 00000100:00080000:0.0:1788575816.667840:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575816.667844:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575816.667847:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575816.667856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570011392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575816.667864:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575816.667867:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575816.667868:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575816.667871:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12570 00000100:00080000:0.0:1788575816.667874:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575816.667877:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788575816.667880:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575816.667882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575816.667891:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570011520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575816.667899:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575816.667901:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575816.667902:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575816.667905:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12570 00000100:00080000:0.0:1788575816.667908:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575816.667910:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575816/real 1788575816] req@ffff920a48353800 x1875444570011136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575816.667911:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575816.667915:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575816.667923:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570011648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575816.667927:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570011136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575816.667941:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575816.667946:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575816.667950:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570011136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575816.667970:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575816.667987:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575816.667996:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575816/real 1788575816] req@ffff920a48350700 x1875444570011264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575816.668008:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570011264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575816.668014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575816.668023:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575816.668027:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575816.668033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575816.668052:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575816.668063:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575816/real 1788575816] req@ffff920a48350380 x1875444570011392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575816.668074:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570011392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575816.668079:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575816.668081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575816.668083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575816.668087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575816.668101:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575816.668107:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575816/real 1788575816] req@ffff920a48353480 x1875444570011520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575816.668113:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570011520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575816.668121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575816.668123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575816.668125:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575816.668128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575816.668136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575816.668141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575816/real 1788575816] req@ffff920a48353b80 x1875444570011648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575816.668146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570011648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575816.668150:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575816.668152:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575816.668153:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575817.115767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575817.691776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575818.139771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575818.715733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575819.163752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575819.739913:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575820.187748:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575820.763701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575821.211775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575821.787697:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575821.787698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575821.787709:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575821.787713:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575821.787725:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12575 00000100:00080000:0.0:1788575821.787736:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575821.787744:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575821.787780:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570011776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575821.787808:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575821.787812:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575821.787814:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575821.787818:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12575 00000100:00080000:0.0:1788575821.787822:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575821.787826:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575821.787835:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575821.787848:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570011904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575821.787859:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575821.787862:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575821.787864:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575821.787870:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12575 00000100:00080000:0.0:1788575821.787873:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575821.787877:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575821.787882:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575821.787885:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570011776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575821.787893:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570012032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575821.787903:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575821.787907:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575821.787909:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575821.787913:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12575 00000100:00080000:0.0:1788575821.787918:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575821.787922:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575821.787928:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575821.787941:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570012160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575821.787950:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788575821.787952:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575821.787957:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575821.787961:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575821.787965:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12575 00000100:00080000:0.0:1788575821.787970:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575821.787975:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575821.787980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575821.787991:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575821/real 1788575821] req@ffff920a48350a80 x1875444570011776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575821.787995:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570012288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575821.788010:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570011776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575821.788026:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575821.788030:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575821.788034:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570011776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575821.788058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575821.788077:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575821.788086:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575821/real 1788575821] req@ffff920a48351500 x1875444570011904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575821.788094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570011904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575821.788099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575821.788101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575821.788103:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570011904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575821.788107:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575821.788117:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575821.788122:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575821/real 1788575821] req@ffff920a48351c00 x1875444570012032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575821.788128:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570012032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575821.788142:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575821.788148:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575821.788151:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575821.788157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575821.788176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575821.788186:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575821/real 1788575821] req@ffff920a48353b80 x1875444570012160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575821.788197:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570012160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575821.788202:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575821.788204:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575821.788205:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575821.788209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575821.788222:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575821.788227:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575821/real 1788575821] req@ffff920a48353480 x1875444570012288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575821.788234:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570012288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575821.788238:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575821.788239:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575821.788241:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575822.235768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575822.811776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575823.259756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575823.835719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575824.283788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575824.859647:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575825.307745:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575825.883725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575826.331822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575826.907688:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575826.907698:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575826.907702:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575826.907712:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12580 00000100:00080000:0.0:1788575826.907723:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575826.907731:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575826.907765:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570012416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575826.907790:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575826.907793:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575826.907795:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575826.907799:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12580 00000100:00080000:0.0:1788575826.907803:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575826.907806:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575826.907815:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575826.907826:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788575826.907827:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570012544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575826.907835:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575826.907838:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575826.907840:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575826.907844:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12580 00000100:00080000:0.0:1788575826.907846:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575826.907850:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575826.907854:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575826.907863:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570012672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575826.907872:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575826.907874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575826.907876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575826.907879:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12580 00000100:00080000:0.0:1788575826.907882:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575826.907882:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570012416:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575826.907886:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575826.907890:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575826.907901:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570012800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575826.907913:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575826.907916:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575826.907918:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575826.907920:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12580 00000100:00080000:0.0:1788575826.907923:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575826.907925:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575826.907929:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575826.907936:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570012928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575826.907950:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575826.907983:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575826/real 1788575826] req@ffff920a48350380 x1875444570012416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575826.907994:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570012416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575826.908000:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575826.908002:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575826.908005:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570012416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575826.908009:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575826.908024:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575826.908031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575826/real 1788575826] req@ffff920a48350700 x1875444570012544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575826.908038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570012544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575826.908042:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575826.908044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575826.908046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575826.908051:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575826.908066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575826.908076:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575826/real 1788575826] req@ffff920a48353800 x1875444570012672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575826.908086:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570012672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575826.908093:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575826.908096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575826.908099:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575826.908126:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575826.908140:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575826.908147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575826/real 1788575826] req@ffff920a48353480 x1875444570012800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575826.908154:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570012800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575826.908158:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575826.908160:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575826.908162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575826.908165:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575826.908173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575826.908178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575826/real 1788575826] req@ffff920a48353b80 x1875444570012928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575826.908198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570012928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575826.908203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575826.908205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575826.908207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570012928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575827.355839:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575827.932320:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575828.379775:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575828.955714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575829.403756:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575829.979694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575830.427803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575831.003724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575831.451808:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575832.027707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575832.027741:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575832.027747:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575832.027752:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575832.027765:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12585 00000100:00080000:1.0:1788575832.027777:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575832.027788:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575832.027837:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570013056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575832.027865:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575832.027867:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575832.027868:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575832.027871:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12585 00000100:00080000:1.0:1788575832.027874:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575832.027878:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575832.027892:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575832.027904:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570013056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575832.027909:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570013184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575832.027921:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575832.027924:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575832.027925:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575832.027928:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12585 00000100:00080000:1.0:1788575832.027931:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575832.027934:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575832.027938:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575832.027947:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570013312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575832.027953:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575832.027955:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575832.027957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575832.027960:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12585 00000100:00080000:1.0:1788575832.027963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575832.027965:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575832.027968:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575832.027972:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575832.027979:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570013440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575832.027986:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575832.027989:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575832.027990:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575832.028001:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12585 00000100:00080000:1.0:1788575832.028004:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575832.028008:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575832.028009:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff920a52890380 x1875444570013056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575832.028014:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575832.028025:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570013568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575832.028068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570013056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575832.028080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575832.028084:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575832.028088:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570013056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575832.028112:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575832.028130:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575832.028138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff920a52891f80 x1875444570013184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575832.028146:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570013184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575832.028156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575832.028158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575832.028160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013184:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575832.028163:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575832.028173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575832.028179:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff920a52891880 x1875444570013312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575832.028185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570013312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575832.028190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575832.028192:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575832.028193:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575832.028196:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575832.028207:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575832.028212:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff920a52893480 x1875444570013440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575832.028219:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570013440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575832.028223:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575832.028224:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575832.028227:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575832.028230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575832.028239:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575832.028244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575832/real 1788575832] req@ffff920a52891500 x1875444570013568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575832.028250:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570013568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575832.028255:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575832.028256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575832.028258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575832.475753:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575833.051697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575833.499781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575834.075697:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575834.523870:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575835.099688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575835.547771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575836.123739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575836.571805:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575837.147723:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575837.147723:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575837.147736:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575837.147741:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575837.147753:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12590 00000100:00080000:1.0:1788575837.147765:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575837.147774:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575837.147823:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570013696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575837.147852:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575837.147856:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575837.147858:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575837.147862:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12590 00000100:00080000:1.0:1788575837.147866:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575837.147871:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575837.147883:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575837.147891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570013696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575837.147914:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570013824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575837.147925:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575837.147930:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575837.147931:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575837.147936:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12590 00000100:00080000:1.0:1788575837.147939:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575837.147944:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575837.147949:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575837.147961:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570013952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575837.147968:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575837.147970:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575837.147972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575837.147975:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12590 00000100:00080000:1.0:1788575837.147979:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575837.147983:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575837.147989:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575837.148002:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570014080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575837.148006:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575837.148017:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575837.148020:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575837.148023:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575837.148028:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12590 00000100:00080000:1.0:1788575837.148032:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575837.148037:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575837.148042:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575837.148055:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff920a52890e00 x1875444570013696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575837.148056:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570014208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575837.148102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570013696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575837.148114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575837.148119:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575837.148123:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570013696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575837.148150:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575837.148180:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575837.148193:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff920a52893800 x1875444570013824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575837.148205:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570013824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575837.148212:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575837.148215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575837.148218:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575837.148224:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575837.148241:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575837.148251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff920a52892d80 x1875444570013952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575837.148261:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570013952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575837.148268:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575837.148271:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575837.148274:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570013952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575837.148279:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575837.148299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575837.148309:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff920a52892300 x1875444570014080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575837.148319:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570014080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575837.148326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575837.148329:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575837.148332:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575837.148337:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575837.148355:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575837.148365:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575837/real 1788575837] req@ffff920a52891180 x1875444570014208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575837.148375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570014208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575837.148382:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575837.148385:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575837.148388:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575837.595857:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575838.171741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575838.619764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575839.196089:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575839.644029:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575840.220506:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575840.667780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575841.243776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575841.691750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575842.267700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575842.267906:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575842.267912:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575842.267916:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575842.267924:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12595 00000100:00080000:0.0:1788575842.267934:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575842.267941:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575842.267977:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570014336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575842.268003:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575842.268006:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575842.268008:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575842.268012:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12595 00000100:00080000:0.0:1788575842.268015:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575842.268019:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575842.268031:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575842.268043:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570014464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575842.268048:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570014336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575842.268056:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575842.268061:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575842.268063:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575842.268067:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12595 00000100:00080000:0.0:1788575842.268071:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575842.268075:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575842.268080:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575842.268091:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570014592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575842.268099:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575842.268102:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575842.268104:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575842.268108:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12595 00000100:00080000:0.0:1788575842.268111:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575842.268115:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575842.268120:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575842.268131:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570014720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575842.268140:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575842.268143:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575842.268145:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575842.268149:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12595 00000100:00080000:0.0:1788575842.268153:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575842.268157:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575842.268162:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575842.268171:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575842.268174:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570014848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575842.268210:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff920a48351c00 x1875444570014336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575842.268228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570014336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575842.268237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575842.268241:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575842.268244:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570014336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575842.268264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575842.268284:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575842.268294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff920a48351500 x1875444570014464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575842.268306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570014464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575842.268313:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575842.268317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575842.268320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575842.268325:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575842.268343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575842.268353:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff920a48350a80 x1875444570014592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575842.268368:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570014592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575842.268375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575842.268377:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575842.268380:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575842.268385:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575842.268400:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575842.268408:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff920a48353b80 x1875444570014720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575842.268418:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570014720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575842.268425:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575842.268428:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575842.268431:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575842.268436:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575842.268451:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575842.268459:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575842/real 1788575842] req@ffff920a48353480 x1875444570014848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575842.268469:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570014848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575842.268481:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575842.268484:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575842.268487:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570014848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575842.717892:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575843.292787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575843.739777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575844.315682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575844.764858:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575845.339729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575845.787822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575846.363769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575846.811867:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575847.387754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575847.387832:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.387839:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575847.387842:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.387854:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12600 00000100:00080000:3.0:1788575847.387865:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.387874:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575847.387980:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570014976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575847.388065:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570014976:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575847.388152:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575847.388190:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.388195:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575847.388197:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.388213:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12600 00000100:00080000:3.0:1788575847.388220:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.388225:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575847.388237:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575847.388252:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570015104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575847.392105:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff920a529b2d80 x1875444570014976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575847.392148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570014976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575847.392160:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575847.392166:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575847.392170:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570014976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575847.392207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575847.392266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575847.392285:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff920a529b3480 x1875444570015104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575847.392297:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570015104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575847.392306:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575847.392310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575847.392313:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015104:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575847.392632:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.392638:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575847.392642:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.392649:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12600 00000100:00080000:3.0:1788575847.392654:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.392662:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575847.392675:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575847.392702:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570015232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.392719:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.392723:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575847.392724:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.392729:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12600 00000100:00080000:3.0:1788575847.392732:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.392737:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575847.392743:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575847.392756:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570015360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575847.392767:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575847.392771:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575847.392773:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575847.392817:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12600 00000100:00080000:3.0:1788575847.392823:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575847.392828:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575847.392835:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575847.392849:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570015488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575847.392990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575847.393016:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575847.393029:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff920a529b2a00 x1875444570015232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575847.393042:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570015232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575847.393051:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575847.393054:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575847.393058:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575847.393064:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575847.393084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575847.393095:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff920a529b1c00 x1875444570015360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575847.393106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570015360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575847.393114:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575847.393117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575847.393120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575847.393126:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575847.393145:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575847.393156:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575847/real 1788575847] req@ffff920a529b3480 x1875444570015488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575847.393167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570015488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575847.393174:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575847.393177:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575847.393181:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575847.836236:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575848.412239:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575848.860101:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575849.435729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575849.884265:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575850.459690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575850.907734:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575851.483705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575851.931744:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575852.507728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575852.507832:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575852.507839:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575852.507843:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575852.507852:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12605 00000100:00080000:0.0:1788575852.507861:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575852.507869:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575852.507913:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570015616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575852.507937:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575852.507941:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575852.507943:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575852.507948:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12605 00000100:00080000:0.0:1788575852.507952:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575852.507956:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575852.507966:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575852.507975:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570015616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575852.507995:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570015744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575852.508013:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575852.508016:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575852.508019:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575852.508023:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12605 00000100:00080000:0.0:1788575852.508026:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575852.508030:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575852.508036:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1788575852.508042:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575852.508048:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570015872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575852.508058:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575852.508062:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575852.508064:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575852.508068:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12605 00000100:00080000:0.0:1788575852.508072:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575852.508077:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575852.508081:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff920a48353800 x1875444570015616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1788575852.508084:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575852.508102:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570016000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575852.508112:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575852.508115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570015616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575852.508118:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575852.508123:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575852.508132:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12605 00000100:00080000:2.0:1788575852.508133:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575852.508140:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575852.508144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570015616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575852.508149:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575852.508156:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575852.508161:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575852.508174:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575852.508175:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570016128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575852.508219:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575852.508233:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff920a48350700 x1875444570015744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575852.508245:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570015744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575852.508252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575852.508255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575852.508258:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575852.508264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575852.508281:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575852.508292:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff920a48350380 x1875444570015872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575852.508302:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570015872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575852.508310:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575852.508313:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575852.508316:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570015872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575852.508322:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575852.508341:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575852.508351:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff920a48353480 x1875444570016000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575852.508359:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570016000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575852.508364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575852.508366:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575852.508368:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575852.508377:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575852.508388:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575852.508394:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575852/real 1788575852] req@ffff920a48353b80 x1875444570016128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575852.508400:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570016128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575852.508405:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575852.508406:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575852.508408:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575852.955781:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575853.531988:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575853.979785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575854.556020:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575855.003761:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575855.579701:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575856.027753:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575856.604602:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575857.052020:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575857.627833:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575857.628610:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575857.628616:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575857.628620:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575857.628629:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12611 00000100:00080000:3.0:1788575857.628638:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575857.628645:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575857.628679:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570016256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575857.628708:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575857.628712:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575857.628714:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575857.628718:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12611 00000100:00080000:3.0:1788575857.628721:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575857.628725:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575857.628734:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575857.628745:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570016384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575857.628754:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575857.628757:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575857.628759:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575857.628762:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12611 00000100:00080000:3.0:1788575857.628765:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575857.628768:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575857.628772:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575857.628781:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570016512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575857.628789:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575857.628792:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575857.628793:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575857.628796:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12611 00000100:00080000:3.0:1788575857.628799:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575857.628803:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575857.628807:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575857.628817:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570016640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575857.628826:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575857.628829:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575857.628831:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575857.628834:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12611 00000100:00080000:3.0:1788575857.628837:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575857.628841:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575857.628845:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575857.628855:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570016768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575857.628941:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570016256:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575857.629005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575857.629037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575857/real 1788575857] req@ffff920a529b2d80 x1875444570016256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575857.629053:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570016256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575857.629062:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575857.629065:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575857.629069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570016256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575857.629075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575857.629097:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575857.629109:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575857/real 1788575857] req@ffff920a529b3100 x1875444570016384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575857.629120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570016384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575857.629127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575857.629130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575857.629133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575857.629138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575857.629156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575857.629166:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575857/real 1788575857] req@ffff920a529b1f80 x1875444570016512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575857.629183:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570016512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575857.629190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575857.629193:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575857.629196:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575857.629202:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575857.629221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575857.629231:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575857/real 1788575857] req@ffff920a529b0000 x1875444570016640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575857.629241:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570016640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575857.629249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575857.629252:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575857.629255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575857.629260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575857.629275:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575857.629283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575857/real 1788575857] req@ffff920a529b0700 x1875444570016768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575857.629292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570016768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575857.629305:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575857.629307:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575857.629310:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570016768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575858.075767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575858.651710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575859.099786:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575859.675655:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575860.123750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575860.699984:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575861.147824:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575861.723693:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575862.171734:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575862.747757:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575862.747763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575862.747771:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575862.747777:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575862.747789:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12616 00000100:00080000:1.0:1788575862.747802:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575862.747810:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575862.747860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444570016896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575862.747897:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575862.747901:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575862.747903:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575862.747907:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12616 00000100:00080000:1.0:1788575862.747910:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575862.747914:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575862.747926:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575862.747938:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444570017024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575862.747947:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575862.747950:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575862.747952:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575862.747964:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12616 00000100:00080000:1.0:1788575862.747968:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575862.747971:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575862.747976:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575862.747989:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570017152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575862.747998:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575862.748001:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575862.748003:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575862.748006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12616 00000100:00080000:1.0:1788575862.748009:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575862.748013:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575862.748018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575862.748027:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570017280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575862.748035:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575862.748038:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575862.748040:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575862.748043:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12616 00000100:00080000:1.0:1788575862.748046:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575862.748050:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575862.748054:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575862.748062:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570017408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575862.748161:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570016896:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575862.748233:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575862.748274:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575862/real 1788575862] req@ffff920a52893100 x1875444570016896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575862.748292:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444570016896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575862.748303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575862.748306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575862.748323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570016896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575862.748330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575862.748359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575862.748373:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575862/real 1788575862] req@ffff920a52890700 x1875444570017024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575862.748390:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444570017024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575862.748399:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575862.748402:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575862.748405:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575862.748411:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575862.748431:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575862.748443:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575862/real 1788575862] req@ffff920a52891180 x1875444570017152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575862.748454:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570017152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575862.748462:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575862.748465:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575862.748468:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575862.748474:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575862.748493:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575862.748502:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575862/real 1788575862] req@ffff920a52892300 x1875444570017280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575862.748512:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570017280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575862.748526:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575862.748530:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575862.748534:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575862.748542:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575862.748560:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575862.748572:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575862/real 1788575862] req@ffff920a52892d80 x1875444570017408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575862.748684:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570017408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575862.748693:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575862.748696:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575862.748699:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575863.195776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575863.771761:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575864.219823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575864.795769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575865.243946:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575865.819956:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575866.267757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575866.843743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575867.291769:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575867.867671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575867.867674:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575867.867682:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575867.867686:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575867.867696:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12621 00000100:00080000:1.0:1788575867.867706:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575867.867713:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575867.867744:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570017536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575867.867766:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575867.867769:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575867.867770:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575867.867773:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12621 00000100:00080000:1.0:1788575867.867776:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575867.867779:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575867.867786:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575867.867794:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570017664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575867.867802:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575867.867805:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575867.867805:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570017536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575867.867809:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575867.867814:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12621 00000100:00080000:1.0:1788575867.867816:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575867.867818:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575867.867821:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575867.867828:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570017792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575867.867834:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575867.867836:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575867.867837:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575867.867839:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12621 00000100:00080000:1.0:1788575867.867841:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575867.867843:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575867.867846:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575867.867852:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570017920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575867.867857:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575867.867859:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575867.867860:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575867.867862:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12621 00000100:00080000:1.0:1788575867.867865:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575867.867867:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575867.867869:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575867.867875:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575867.867878:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570018048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575867.867916:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575867/real 1788575867] req@ffff920a52893800 x1875444570017536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575867.867935:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570017536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575867.867945:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575867.867949:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575867.867953:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570017536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575867.867973:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575867.867994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575867.868007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575867/real 1788575867] req@ffff920a52890e00 x1875444570017664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575867.868018:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570017664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575867.868026:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575867.868029:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575867.868033:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575867.868037:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575867.868051:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575867.868058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575867/real 1788575867] req@ffff920a52891500 x1875444570017792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575867.868064:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570017792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575867.868073:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575867.868081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575867.868084:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575867.868090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575867.868113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575867.868124:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575867/real 1788575867] req@ffff920a52893480 x1875444570017920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575867.868133:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570017920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575867.868138:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575867.868141:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575867.868144:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570017920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575867.868149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575867.868167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575867.868177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575867/real 1788575867] req@ffff920a52891880 x1875444570018048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575867.868188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570018048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575867.868193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575867.868195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575867.868197:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575868.315742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575868.891691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575869.340238:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575869.915676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575870.363722:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575870.940126:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575871.387762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575871.963731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575872.411762:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575872.987703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575872.987712:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575872.987717:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575872.987720:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575872.987729:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12626 00000100:00080000:3.0:1788575872.987737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575872.987744:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575872.987772:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570018176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575872.987791:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575872.987794:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575872.987796:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575872.987799:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12626 00000100:00080000:3.0:1788575872.987802:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575872.987804:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575872.987812:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575872.987819:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570018304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575872.987827:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575872.987829:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575872.987831:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575872.987836:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570018176:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575872.987837:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12626 00000100:00080000:3.0:1788575872.987842:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575872.987846:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575872.987849:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575872.987856:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570018432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575872.987861:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575872.987864:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575872.987865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575872.987867:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12626 00000100:00080000:3.0:1788575872.987869:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575872.987871:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575872.987874:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575872.987880:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570018560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575872.987886:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575872.987889:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575872.987890:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575872.987893:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12626 00000100:00080000:3.0:1788575872.987895:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575872.987898:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575872.987903:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575872.987912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570018688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575872.987945:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575872.987983:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575872.988006:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575872.988024:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575873.435796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575874.011755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575874.459824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575875.035740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575875.483804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575876.059745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788575876.060236:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788575876.060283:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788575876.060301:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575876.060334:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575876.060347:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788575876.060359:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788575876.060360:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575872/real 1788575876] req@ffff920a529b0380 x1875444570018304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1788575876.060372:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788575876.060404:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570018304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575876.060416:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575876.060422:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575876.060426:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575876.060434:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575872/real 1788575876] req@ffff920a529b3b80 x1875444570018432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575876.060446:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570018432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575876.060454:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575876.060458:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575876.060461:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575876.060469:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575872/real 1788575876] req@ffff920a529b1180 x1875444570018560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575876.060502:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570018560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575876.060510:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575876.060514:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575876.060517:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575876.060524:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575872/real 1788575876] req@ffff920a529b1500 x1875444570018688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575876.060535:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570018688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575876.060542:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575876.060545:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575876.060548:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018688:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575876.060555:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575872/real 1788575876] req@ffff920a529b2300 x1875444570018176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575876.060566:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570018176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575897 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575876.060573:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575876.060576:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575876.060580:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570018176:192.168.202.157@tcp:250:kworker.0 00010000:00010000:2.0:1788575876.508153:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575877.083686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575877.531886:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575878.107906:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575878.110334:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.110344:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575878.110348:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.110358:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12631 00000100:00080000:1.0:1788575878.110370:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.110378:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575878.110414:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570018816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.110448:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.110452:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.110454:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.110459:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12631 00000100:00080000:1.0:1788575878.110462:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.110467:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575878.110477:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575878.110488:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570018944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.110498:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.110501:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575878.110503:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.110507:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12631 00000100:00080000:1.0:1788575878.110511:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.110515:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575878.110520:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575878.110529:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570019072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.110538:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.110542:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575878.110544:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.110547:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12631 00000100:00080000:1.0:1788575878.110551:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.110555:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575878.110560:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575878.110584:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570018816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575878.110607:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570019200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575878.110618:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575878.110621:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575878.110623:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575878.110627:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12631 00000100:00080000:1.0:1788575878.110631:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575878.110635:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575878.110639:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575878.110660:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570019328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788575878.110960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575878.111007:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff920a52891f80 x1875444570018816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.111083:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570018816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.111094:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.111098:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575878.111101:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570018816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575878.111109:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575878.111134:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575878.111147:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff920a52890380 x1875444570018944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.111158:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570018944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.111166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.111169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575878.111172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570018944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575878.111178:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575878.111196:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575878.111208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff920a52893b80 x1875444570019072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.111225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570019072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.111232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.111235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575878.111238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575878.111265:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575878.111286:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575878.111298:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff920a52890000 x1875444570019200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.111307:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570019200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.111314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.111317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575878.111320:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575878.111326:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575878.111344:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575878.111354:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575878/real 1788575878] req@ffff920a52892680 x1875444570019328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575878.111364:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570019328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575878.111794:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575878.111799:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575878.111803:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575878.555920:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575879.131679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575879.579825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575880.155715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575880.603812:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575881.179972:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575881.627910:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575882.203715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575882.652611:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575883.227732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575883.228114:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.228120:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575883.228124:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.228134:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12636 00000100:00080000:0.0:1788575883.228144:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.228151:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575883.228184:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570019456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.228212:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.228216:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575883.228218:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.228223:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12636 00000100:00080000:0.0:1788575883.228227:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.228232:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575883.228242:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575883.228252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570019456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575883.228258:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570019584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.228269:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575883.228274:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575883.228277:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.228281:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12636 00000100:00080000:0.0:1788575883.228285:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.228289:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575883.228295:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575883.228308:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570019712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575883.228317:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575883.228319:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575883.228323:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575883.228327:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.228333:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12636 00000100:00080000:0.0:1788575883.228350:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575883.228356:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff920a48350a80 x1875444570019456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575883.228357:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575883.228366:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575883.228384:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570019456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575883.228386:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570019840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575883.228407:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575883.228416:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00080000:0.0:1788575883.228417:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575883.228421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570019456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575883.228431:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575883.228434:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575883.228439:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12636 00000100:00080000:0.0:1788575883.228443:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575883.228448:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575883.228451:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019584:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788575883.228455:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575883.228472:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570019968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575883.228483:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575883.228500:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff920a48351500 x1875444570019584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575883.228512:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570019584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575883.228520:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575883.228523:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575883.228527:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575883.228533:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575883.228553:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575883.228566:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff920a48351c00 x1875444570019712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575883.228579:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570019712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575883.228611:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575883.228615:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575883.228618:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575883.228624:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575883.228646:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575883.228657:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff920a48353b80 x1875444570019840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575883.228667:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570019840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575883.228674:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575883.228676:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575883.228679:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575883.228695:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575883.228715:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575883.228728:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575883/real 1788575883] req@ffff920a48353480 x1875444570019968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575883.228739:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570019968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575883.228753:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575883.228757:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575883.228760:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570019968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575883.675795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575884.251730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575884.701030:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575885.275716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575885.723766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575886.299698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575886.747787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575887.323740:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575887.771730:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575888.347728:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575888.347736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575888.347740:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575888.347745:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575888.347761:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12641 00000100:00080000:0.0:1788575888.347775:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575888.347794:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575888.347947:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570020096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575888.348020:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575888.348024:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575888.348026:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575888.348030:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12641 00000100:00100000:1.0:1788575888.348060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570020096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575888.348062:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575888.348092:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575888.348107:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575888.348150:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570020224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575888.348155:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575888.348185:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575888.348189:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575888.348191:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1788575888.348196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff920a48350380 x1875444570020096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.348198:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12641 00000100:00100000:1.0:1788575888.348222:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570020096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.348233:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575888.348236:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575888.348246:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:1.0:1788575888.348246:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575888.348257:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570020096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575888.348277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020224:192.168.202.157@tcp:38:kworker.0 00000008:00100000:0.0:1788575888.348285:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575888.348301:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570020352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1788575888.348313:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575888.348323:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff920a48350700 x1875444570020224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.348342:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575888.348348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570020224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.348350:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575888.348355:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575888.348361:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12641 00000100:00080000:0.0:1788575888.348367:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575888.348376:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575888.348385:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:1.0:1788575888.348391:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575888.348402:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575888.348406:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575888.348418:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575888.348441:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575888.348448:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570020480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575888.348453:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff920a48353800 x1875444570020352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575888.348469:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570020352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575888.348474:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575888.348476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575888.348478:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020352:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575888.348511:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575888.348516:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575888.348518:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575888.348523:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12641 00000100:00080000:0.0:1788575888.348528:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788575888.348534:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575888.348552:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575888.348559:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff920a48350700 x1875444570020480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575888.348562:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575888.348570:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575888.348613:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570020480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575888.348622:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575888.348624:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575888.348627:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575888.348645:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570020608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575888.348686:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575888.348711:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575888.348728:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575888/real 1788575888] req@ffff920a48350380 x1875444570020608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575888.348739:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570020608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575888.348745:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575888.348747:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575888.348749:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575888.795771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575889.371689:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575889.819790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575890.400028:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575890.843788:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575891.419850:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575891.868301:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575892.443670:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575892.891871:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575893.467712:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575893.467730:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575893.467734:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575893.467739:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575893.467750:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12646 00000100:00080000:3.0:1788575893.467760:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575893.467768:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575893.467803:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570020736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575893.467827:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575893.467831:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575893.467832:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575893.467838:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12646 00000100:00080000:3.0:1788575893.467842:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575893.467845:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575893.467855:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575893.467868:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570020864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575893.467877:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575893.467880:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575893.467882:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575893.467886:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12646 00000100:00080000:3.0:1788575893.467888:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575893.467892:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575893.467896:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575893.467905:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570020992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575893.467914:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575893.467917:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575893.467918:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575893.467922:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12646 00000100:00080000:3.0:1788575893.467925:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575893.467929:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575893.467934:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575893.467943:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570021120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575893.467952:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575893.467955:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575893.467957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575893.467960:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12646 00000100:00080000:3.0:1788575893.467963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575893.467966:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575893.467970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575893.467980:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570021248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788575893.468099:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570020736:192.168.202.157@tcp:250:kworker.0 00000800:00000100:1.0:1788575893.468165:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575893.468196:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff920a529b0e00 x1875444570020736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.468211:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570020736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.468221:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.468225:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575893.468229:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570020736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575893.468235:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575893.468261:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575893.468272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff920a529b2300 x1875444570020864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.468283:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570020864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.468291:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.468294:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575893.468296:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575893.468301:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575893.468319:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575893.468328:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff920a529b1500 x1875444570020992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.468339:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570020992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.468346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.468349:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575893.468352:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570020992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575893.468357:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575893.468372:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575893.468380:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff920a529b1180 x1875444570021120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.468396:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570021120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.468402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.468405:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575893.468408:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575893.468412:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575893.468431:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575893.468440:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575893/real 1788575893] req@ffff920a529b3b80 x1875444570021248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575893.468449:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570021248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575893.468456:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575893.468458:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575893.468461:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575893.915780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575894.491834:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575894.939776:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575895.515727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575895.963773:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575896.539667:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575896.989760:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575897.564203:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575898.011874:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575898.587688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575898.587962:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.587969:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575898.587972:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.587993:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12651 00000100:00080000:0.0:1788575898.588002:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.588010:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575898.588043:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570021376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.588066:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.588070:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575898.588072:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.588076:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12651 00000100:00080000:0.0:1788575898.588080:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.588084:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575898.588093:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575898.588101:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570021376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575898.588107:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570021504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.588117:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.588121:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575898.588122:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.588126:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12651 00000100:00080000:0.0:1788575898.588129:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.588133:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575898.588137:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575898.588148:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570021632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.588158:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1788575898.588164:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575898.588165:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575898.588169:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.588173:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12651 00000100:00080000:0.0:1788575898.588178:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.588181:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575898.588186:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575898.588195:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570021760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575898.588204:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575898.588209:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1788575898.588209:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff920a48353480 x1875444570021376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575898.588211:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575898.588218:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12651 00000100:00080000:0.0:1788575898.588224:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575898.588230:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575898.588237:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575898.588252:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570021376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575898.588253:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570021888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575898.588284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.588289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575898.588293:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570021376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788575898.588314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575898.588334:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575898.588345:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff920a48353b80 x1875444570021504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.588356:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570021504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.588364:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.588367:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575898.588371:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575898.588376:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788575898.588390:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575898.588398:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff920a48351c00 x1875444570021632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.588409:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570021632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.588422:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.588425:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575898.588428:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788575898.588433:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575898.588447:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575898.588455:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff920a48351500 x1875444570021760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.588465:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570021760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.588472:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.588474:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575898.588477:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788575898.588482:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788575898.588496:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575898.588504:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575898/real 1788575898] req@ffff920a48350a80 x1875444570021888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788575898.588514:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570021888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788575898.588520:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788575898.588523:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788575898.588527:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570021888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575899.035818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575899.611692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575900.061616:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575900.635832:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575901.083748:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575901.659695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575902.108072:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575902.683826:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575903.133050:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575903.707811:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575903.707822:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575903.707828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575903.707841:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12657 00000100:00080000:3.0:1788575903.707853:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575903.707863:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575903.707912:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570022016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575903.707980:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570022016:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788575903.708055:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575903.708087:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575903.708090:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575903.708092:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575903.708095:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12657 00000100:00080000:3.0:1788575903.708098:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575903.708101:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575903.708111:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575903.708121:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570022144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575903.708223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575903/real 1788575903] req@ffff920a529b0380 x1875444570022016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575903.708240:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570022016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575903.708248:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575903.708251:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575903.708255:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570022016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575903.708277:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575903.708295:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575903.708303:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575903/real 1788575903] req@ffff920a529b0700 x1875444570022144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575903.708311:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570022144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575903.708315:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575903.708317:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575903.708319:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022144:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575903.708515:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575903.708519:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575903.708521:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575903.708526:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12657 00000100:00080000:3.0:1788575903.708530:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575903.708534:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575903.708539:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575903.708552:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570022272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575903.708605:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575903.708629:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575903.708684:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575903.708688:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575903.708690:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575903.708695:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12657 00000100:00080000:3.0:1788575903.708699:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575903.708703:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575903.708709:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575903.708721:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570022400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575903.708759:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575903/real 1788575903] req@ffff920a529b0000 x1875444570022272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575903.708768:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570022272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575903.708778:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575903.708780:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575903.708782:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575903.708790:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575903.708806:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575903.708812:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575903/real 1788575903] req@ffff920a529b1f80 x1875444570022400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575903.708819:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570022400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575903.708824:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575903.708825:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575903.708827:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022400:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788575903.708931:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575903.708973:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575903.708975:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575903.708979:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12657 00000100:00080000:3.0:1788575903.708982:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575903.708985:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575903.708989:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575903.708998:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570022528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575903.709039:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575903.709062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575903.709127:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575903/real 1788575903] req@ffff920a529b3100 x1875444570022528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575903.709139:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570022528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575903.709144:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575903.709146:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575903.709148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022528:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575903.714310:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575904.156015:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575904.731676:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575905.183515:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575905.756663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575906.204315:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575906.779724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575907.227798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575907.803751:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575908.252538:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788575908.827723:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575908.827734:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575908.827739:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575908.827750:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12662 00000100:00080000:3.0:1788575908.827762:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575908.827773:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575908.827817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570022656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575908.827843:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575908.827847:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575908.827848:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575908.827852:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12662 00000100:00080000:3.0:1788575908.827854:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575908.827867:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575908.827877:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575908.827887:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570022784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575908.827901:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575908.827904:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575908.827905:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575908.827909:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12662 00000100:00080000:3.0:1788575908.827912:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575908.827915:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575908.827918:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575908.827925:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570022912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575908.827932:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575908.827935:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575908.827937:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575908.827946:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12662 00000100:00080000:3.0:1788575908.827949:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575908.827952:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575908.827956:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575908.827963:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570023040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575908.827970:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575908.827973:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575908.827974:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575908.827977:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12662 00000100:00080000:3.0:1788575908.827980:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575908.827982:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575908.827986:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575908.827994:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570023168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575908.828990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570022656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575908.829006:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000800:00000100:0.0:1788575908.829065:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575908.829103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575908/real 1788575908] req@ffff920a529b2d80 x1875444570022656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575908.829122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570022656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575908.829134:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575908.829138:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575908.829142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570022656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575908.829149:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575908.829167:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575908.829177:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575908/real 1788575908] req@ffff920a529b3480 x1875444570022784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575908.829188:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570022784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575908.829197:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575908.829200:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575908.829203:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575908.829218:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575908.829235:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575908.829244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575908/real 1788575908] req@ffff920a529b1c00 x1875444570022912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575908.829256:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570022912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575908.829263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575908.829267:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575908.829270:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570022912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575908.829275:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575908.829291:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575908.829299:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575908/real 1788575908] req@ffff920a529b2a00 x1875444570023040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575908.829310:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570023040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575908.829317:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575908.829321:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575908.829323:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575908.829329:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575908.829343:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575908.829352:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575908/real 1788575908] req@ffff920a529b3100 x1875444570023168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575908.829372:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570023168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575908.829380:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575908.829382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575908.829386:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575909.275782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575909.851722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575910.299843:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575910.875767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575911.323780:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575911.899776:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575912.347808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575912.923733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575913.371787:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575913.947743:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575913.948037:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575913.948043:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575913.948048:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575913.948059:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12667 00000100:00080000:1.0:1788575913.948070:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575913.948079:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575913.948131:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444570023296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575913.948167:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575913.948170:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575913.948172:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575913.948177:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12667 00000100:00080000:1.0:1788575913.948180:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575913.948184:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575913.948196:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575913.948209:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444570023424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575913.948223:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575913.948226:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575913.948227:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575913.948231:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12667 00000100:00080000:1.0:1788575913.948235:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575913.948238:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575913.948243:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575913.948255:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570023552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575913.948262:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575913.948265:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575913.948267:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575913.948270:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12667 00000100:00080000:1.0:1788575913.948273:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575913.948276:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575913.948281:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575913.948289:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570023680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575913.948297:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575913.948478:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570023296:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575913.948559:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788575913.948986:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575913.948989:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575913.948993:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12667 00000100:00080000:1.0:1788575913.948998:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575913.949002:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575913.949007:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575913.949020:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570023808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575913.949091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575913/real 1788575913] req@ffff920a52892a00 x1875444570023296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575913.949110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444570023296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575913.949121:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575913.949125:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575913.949128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570023296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575913.949136:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575913.949161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575913.949175:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575913/real 1788575913] req@ffff920a52890a80 x1875444570023424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575913.949186:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444570023424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575913.949194:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575913.949197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575913.949200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575913.949205:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575913.949225:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575913.949235:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575913/real 1788575913] req@ffff920a52892680 x1875444570023552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575913.949246:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570023552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575913.949253:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575913.949256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575913.949259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575913.949264:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575913.949284:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575913.949294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575913/real 1788575913] req@ffff920a52890000 x1875444570023680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575913.949498:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570023680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575913.949511:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575913.949514:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575913.949517:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575913.949547:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575913.949570:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575913.949582:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575913/real 1788575913] req@ffff920a52893b80 x1875444570023808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575913.949610:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570023808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575938 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575913.949617:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575913.949620:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575913.949623:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570023808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575914.395796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575914.971782:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575915.419900:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575915.995709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575916.443767:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575917.019707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575917.467826:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575918.043719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575918.491806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575919.067725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575919.067725:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.067738:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575919.067743:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.067764:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12672 00000100:00080000:1.0:1788575919.067777:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.067787:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575919.067844:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570023936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.067873:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.067877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575919.067879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.067883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12672 00000100:00080000:1.0:1788575919.067886:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.067891:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575919.067904:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575919.067918:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570024064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.067930:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.067936:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788575919.067939:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570023936:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575919.067940:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.067947:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12672 00000100:00080000:1.0:1788575919.067955:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.067960:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575919.067965:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575919.067977:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570024192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.067993:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.067997:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575919.067998:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.068003:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12672 00000100:00080000:1.0:1788575919.068008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.068012:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575919.068018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575919.068018:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575919.068038:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570024320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575919.068046:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575919.068048:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575919.068050:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575919.068054:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12672 00000100:00080000:1.0:1788575919.068061:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575919.068065:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575919.068073:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff920a52890380 x1875444570023936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1788575919.068074:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575919.068087:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570024448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575919.068124:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570023936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575919.068140:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575919.068144:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575919.068147:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570023936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575919.068169:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575919.068198:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575919.068211:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff920a52891f80 x1875444570024064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575919.068223:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570024064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575919.068229:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575919.068231:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575919.068237:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575919.068244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575919.068287:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575919.068313:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff920a52891880 x1875444570024192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575919.068329:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570024192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575919.068337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575919.068340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575919.068344:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575919.068353:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575919.068383:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575919.068400:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff920a52893480 x1875444570024320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575919.068408:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570024320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575919.068414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575919.068417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575919.068418:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575919.068423:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575919.068439:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575919.068458:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575919/real 1788575919] req@ffff920a52891500 x1875444570024448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575919.068471:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570024448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575919.068479:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575919.068483:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575919.068486:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575919.515824:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575920.091678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575920.539764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575921.115736:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575921.565093:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575922.139688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575922.587808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575923.163722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575923.611820:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575924.187739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788575924.187741:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575924.187754:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575924.187759:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575924.187770:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12677 00000100:00080000:1.0:1788575924.187790:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575924.187797:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575924.187832:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570024576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575924.187857:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575924.187860:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575924.187861:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575924.187864:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12677 00000100:00080000:1.0:1788575924.187866:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575924.187868:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575924.187875:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575924.187885:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570024704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575924.187897:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575924.187900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575924.187903:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575924.187906:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570024576:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788575924.187912:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12677 00000100:00080000:1.0:1788575924.187923:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575924.187930:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575924.187935:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575924.187948:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570024832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575924.187958:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575924.187962:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575924.187964:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575924.187968:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12677 00000100:00080000:1.0:1788575924.187980:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575924.187987:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575924.187993:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575924.187999:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788575924.188010:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570024960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575924.188023:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575924.188025:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575924.188027:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575924.188034:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12677 00000100:00080000:1.0:1788575924.188038:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575924.188042:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575924.188047:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575924.188058:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570025088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575924.188058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575924] req@ffff920a52890e00 x1875444570024576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575924.188116:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570024576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575924.188130:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575924.188136:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575924.188142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570024576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575924.188171:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575924.188193:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575924.188217:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575924] req@ffff920a52893800 x1875444570024704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575924.188231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570024704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575924.188240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575924.188243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575924.188246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575924.188252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575924.188270:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575924.188279:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575924] req@ffff920a52892d80 x1875444570024832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575924.188296:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570024832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575924.188303:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575924.188306:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575924.188309:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575924.188315:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575924.188331:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575924.188339:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575924] req@ffff920a52892300 x1875444570024960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575924.188350:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570024960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575924.188369:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575924.188373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575924.188376:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570024960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575924.188381:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575924.188399:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575924.188408:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575924/real 1788575924] req@ffff920a52891180 x1875444570025088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575924.188418:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570025088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575924.188432:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575924.188434:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575924.188437:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575924.635799:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575925.211756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575925.660349:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575926.235704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575926.683777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575927.259692:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575927.707737:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575928.283666:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575928.731752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575929.307687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788575929.307705:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575929.307713:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788575929.307717:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575929.307727:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12682 00000100:00080000:3.0:1788575929.307737:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575929.307743:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788575929.307779:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570025216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575929.307804:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575929.307816:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788575929.307819:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575929.307824:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12682 00000100:00080000:3.0:1788575929.307828:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788575929.307833:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575929.307839:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570025216:192.168.202.157@tcp:250:kworker.0 00000008:00100000:3.0:1788575929.307844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575929.307860:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570025344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788575929.307871:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575929.307875:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788575929.307878:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575929.307882:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12682 00000100:00080000:3.0:1788575929.307894:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788575929.307901:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788575929.307908:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575929.307914:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788575929.307927:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570025472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575929.307938:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff920a529b1f80 x1875444570025216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.307939:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575929.307947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788575929.307951:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575929.307960:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12682 00000100:00080000:3.0:1788575929.307965:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575929.307967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570025216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788575929.307973:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575929.307980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575929.307986:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575929.307993:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788575929.307994:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570025600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575929.308000:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570025216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788575929.308013:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788575929.308017:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788575929.308019:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788575929.308024:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12682 00000100:00080000:3.0:1788575929.308028:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788575929.308030:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025344:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788575929.308034:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788575929.308041:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788575929.308053:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570025728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575929.308062:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575929.308079:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff920a529b0000 x1875444570025344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575929.308091:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570025344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575929.308099:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575929.308103:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575929.308106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575929.308111:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575929.308129:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575929.308138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff920a529b0700 x1875444570025472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575929.308149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570025472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575929.308156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575929.308174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575929.308178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575929.308183:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575929.308204:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575929.308213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff920a529b0380 x1875444570025600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575929.308225:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570025600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575929.308232:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575929.308235:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575929.308238:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575929.308252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575929.308266:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575929.308272:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575929/real 1788575929] req@ffff920a529b3b80 x1875444570025728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575929.308279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570025728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575929.308284:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575929.308286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575929.308288:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575929.755763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575930.331703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575930.779764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575931.355691:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575931.803757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575932.379706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575932.827752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575933.403715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575933.854167:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575934.427695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575934.427709:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.427714:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575934.427718:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.427726:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12687 00000100:00080000:2.0:1788575934.427734:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.427740:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575934.427770:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570025856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.427792:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.427795:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575934.427796:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.427799:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12687 00000100:00080000:2.0:1788575934.427800:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.427803:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575934.427811:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575934.427819:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570025984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.427824:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.427827:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575934.427828:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.427832:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12687 00000100:00100000:0.0:1788575934.427835:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570025856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575934.427835:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.427839:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575934.427844:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575934.427851:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570026112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.427856:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.427859:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575934.427860:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575934.427863:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12687 00000100:00080000:2.0:1788575934.427866:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.427869:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575934.427874:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575934.427883:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570026240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575934.427892:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575934.427895:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575934.427897:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788575934.427903:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575934.427903:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12687 00000100:00080000:2.0:1788575934.427909:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575934.427913:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575934.427918:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575934.427930:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570026368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575934.427945:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff920a60ac4000 x1875444570025856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.427961:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570025856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.427976:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.427980:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575934.427984:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570025856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575934.427990:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575934.428011:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575934.428020:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff920a60ac6a00 x1875444570025984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.428027:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570025984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.428032:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.428034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575934.428036:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570025984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575934.428053:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575934.428063:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575934.428069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff920a60ac5180 x1875444570026112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.428075:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570026112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.428080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.428082:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575934.428083:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575934.428087:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575934.428096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575934.428101:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff920a60ac6300 x1875444570026240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.428108:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570026240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.428141:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.428145:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575934.428148:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575934.428154:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575934.428173:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575934.428183:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575934/real 1788575934] req@ffff920a60ac7100 x1875444570026368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575934.428191:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570026368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575934.428196:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575934.428198:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575934.428206:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575934.875762:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575935.451704:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575935.899728:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575936.475742:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575936.923812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575937.499724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575937.947836:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575938.524154:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575938.971796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575939.547740:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575939.547751:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575939.547756:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575939.547766:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12692 00000100:00080000:2.0:1788575939.547776:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575939.547781:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575939.547817:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444570026496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575939.547850:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575939.547853:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575939.547854:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575939.547858:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12692 00000100:00080000:2.0:1788575939.547861:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575939.547863:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575939.547871:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575939.547881:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570026624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575939.547888:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575939.547891:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575939.547892:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575939.547895:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12692 00000100:00080000:2.0:1788575939.547897:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575939.547899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575939.547901:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575939.547909:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570026752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575939.547911:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570026496:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788575939.547920:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575939.547924:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575939.547924:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575939.547927:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12692 00000100:00080000:2.0:1788575939.547928:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575939.547930:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575939.547933:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575939.547940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570026880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575939.547945:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575939.547947:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575939.547948:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575939.547951:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12692 00000100:00080000:2.0:1788575939.547953:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575939.547955:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575939.547957:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575939.547963:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570027008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575939.548015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575939.548062:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575939.548130:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575939.548157:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027008:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575939.548197:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575939.995774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575940.571708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575941.019794:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575941.595686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575942.043750:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575942.619713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:1.0:1788575942.619875:0:2384:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:1.0:1788575942.619917:0:2384:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1788575942.619934:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575942.621408:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575942] req@ffff920a60ac5500 x1875444570026496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575964 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575942.621439:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444570026496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575964 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575942.621450:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575942.621455:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575942.621459:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570026496:192.168.202.157@tcp:250:kworker.0 00000800:00000100:1.0:1788575942.621520:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575942.622106:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575942.622117:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:1.0:1788575942.622125:0:2384:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:2.0:1788575942.622247:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575942] req@ffff920a60ac7100 x1875444570026752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575942.622260:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570026752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575942.622274:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575942.622278:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575942.622282:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575942.622288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575942] req@ffff920a60ac6300 x1875444570026880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575942.622300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570026880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575942.622307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575942.622311:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575942.622314:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575942.622320:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575942] req@ffff920a60ac5180 x1875444570027008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575942.622328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570027008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575942.622333:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575942.622335:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575942.622337:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027008:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575942.622341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575939/real 1788575942] req@ffff920a60ac4a80 x1875444570026624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575942.622347:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570026624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575942.622360:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575942.622362:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575942.622364:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570026624:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788575943.067768:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575943.643705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575944.091823:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575944.667732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575944.667755:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575944.667763:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575944.667768:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575944.667781:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12698 00000100:00080000:0.0:1788575944.667793:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575944.667802:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575944.667852:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570027136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575944.667896:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575944.667900:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575944.667902:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575944.667906:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12698 00000100:00080000:0.0:1788575944.667910:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575944.667915:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575944.667928:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575944.667941:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570027264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575944.667951:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575944.667954:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575944.667957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575944.667961:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12698 00000100:00080000:0.0:1788575944.667965:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575944.667969:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575944.667974:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575944.667989:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570027392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575944.667989:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570027136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575944.668005:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575944.668008:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575944.668009:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575944.668013:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12698 00000100:00080000:0.0:1788575944.668025:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575944.668030:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575944.668044:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575944.668056:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570027520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575944.668075:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575944.668078:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575944.668080:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575944.668083:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12698 00000100:00080000:0.0:1788575944.668087:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575944.668091:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575944.668096:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575944.668108:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570027648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575944.668113:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575944.668161:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575944/real 1788575944] req@ffff920a48350a80 x1875444570027136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575944.668199:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570027136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575944.668211:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575944.668215:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575944.668219:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570027136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575944.668226:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575944.668248:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575944.668261:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575944/real 1788575944] req@ffff920a48351500 x1875444570027264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575944.668271:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570027264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575944.668283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575944.668286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575944.668289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027264:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575944.668329:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575944.668349:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575944.668360:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575944/real 1788575944] req@ffff920a48351c00 x1875444570027392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575944.668371:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570027392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575944.668379:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575944.668382:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575944.668385:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575944.668396:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575944.668415:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575944.668425:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575944/real 1788575944] req@ffff920a48353b80 x1875444570027520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575944.668437:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570027520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575944.668443:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575944.668446:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575944.668449:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575944.668455:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575944.668473:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575944.668483:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575944/real 1788575944] req@ffff920a48353480 x1875444570027648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575944.668494:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570027648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575944.668501:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575944.668504:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575944.668513:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575945.115901:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575945.691794:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575946.139829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575946.715814:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575947.163938:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575947.739771:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575948.187810:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788575948.763811:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575949.211827:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575949.787764:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575949.787780:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575949.787784:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575949.787792:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575949.787804:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12703 00000100:00080000:2.0:1788575949.787818:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575949.787828:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575949.787886:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570027776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575949.787917:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575949.787923:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575949.787925:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575949.787929:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12703 00000100:00080000:2.0:1788575949.787934:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575949.787939:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575949.787951:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575949.787961:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570027776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575949.787966:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570027904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575949.787994:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575949.787997:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575949.787999:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575949.788004:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12703 00000100:00080000:2.0:1788575949.788015:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575949.788019:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575949.788024:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575949.788035:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570028032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575949.788045:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575949.788048:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575949.788051:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575949.788055:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12703 00000100:00080000:2.0:1788575949.788060:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575949.788064:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575949.788070:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788575949.788076:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575949.788083:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570028160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575949.788095:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575949.788097:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575949.788099:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575949.788102:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12703 00000100:00080000:2.0:1788575949.788115:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575949.788119:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575949.788123:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575949.788136:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570028288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575949.788140:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575949/real 1788575949] req@ffff920a60ac4a80 x1875444570027776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575949.788168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570027776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575949.788180:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575949.788184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575949.788188:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570027776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575949.788215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575949.788237:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575949.788250:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575949/real 1788575949] req@ffff920a60ac5180 x1875444570027904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575949.788262:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570027904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575949.788270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575949.788273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575949.788276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570027904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575949.788282:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575949.788313:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575949.788324:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575949/real 1788575949] req@ffff920a60ac6300 x1875444570028032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575949.788335:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570028032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575949.788343:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575949.788346:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575949.788349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575949.788355:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575949.788373:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575949.788385:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575949/real 1788575949] req@ffff920a60ac7100 x1875444570028160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575949.788396:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570028160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575949.788403:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575949.788407:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575949.788410:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575949.788415:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575949.788444:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575949.788459:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575949/real 1788575949] req@ffff920a60ac6a00 x1875444570028288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575949.788477:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570028288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575949.788484:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575949.788486:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575949.788489:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575950.235785:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575950.811722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575951.259772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575951.835808:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575952.283772:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575952.859711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575953.307808:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575953.883733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575954.331775:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575954.907837:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575954.907848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575954.907853:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575954.907864:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12708 00000100:00080000:0.0:1788575954.907874:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575954.907881:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575954.907917:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570028416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575954.907967:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575954.907971:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575954.907973:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575954.907978:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12708 00000100:00080000:0.0:1788575954.907982:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575954.907987:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788575954.907987:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000008:00100000:0.0:1788575954.908001:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575954.908015:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570028416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575954.908018:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570028544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575954.908031:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575954.908035:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575954.908037:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575954.908042:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12708 00000100:00080000:0.0:1788575954.908046:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575954.908051:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575954.908056:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575954.908069:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570028672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575954.908078:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575954.908082:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575954.908087:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575954.908087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575954.908095:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12708 00000100:00080000:0.0:1788575954.908100:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575954.908104:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575954.908110:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575954.908124:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570028800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575954.908134:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575954.908137:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:2.0:1788575954.908139:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575954/real 1788575954] req@ffff920a48350380 x1875444570028416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575954.908141:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575954.908149:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12708 00000100:00080000:0.0:1788575954.908156:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575954.908161:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575954.908167:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575954.908167:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570028416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575954.908182:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1788575954.908185:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570028928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575954.908197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575954.908209:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570028416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575954.908231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575954.908253:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575954.908275:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575954/real 1788575954] req@ffff920a48350700 x1875444570028544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575954.908301:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570028544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575954.908310:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575954.908314:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575954.908317:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575954.908323:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575954.908345:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575954.908356:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575954/real 1788575954] req@ffff920a48353800 x1875444570028672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575954.908368:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570028672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575954.908376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575954.908379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575954.908383:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575954.908388:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575954.908409:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575954.908419:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575954/real 1788575954] req@ffff920a48353480 x1875444570028800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575954.908429:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570028800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575954.908435:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575954.908438:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575954.908441:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575954.908457:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575954.908476:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575954.908490:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575954/real 1788575954] req@ffff920a48353b80 x1875444570028928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575954.908503:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570028928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575954.908511:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575954.908514:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575954.908517:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570028928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575955.355796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575955.931718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575956.379782:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575956.955711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575957.405405:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575957.979715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575958.428068:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575959.003716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575959.451892:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575960.027814:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575960.027984:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.028057:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575960.028061:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.028307:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12713 00000100:00080000:0.0:1788575960.028318:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.028325:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575960.028359:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570029056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575960.028483:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570029056:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575960.028543:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575960.028572:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.028576:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.028578:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.028583:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12713 00000100:00080000:0.0:1788575960.028604:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.028609:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575960.028619:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.028633:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570029184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575960.028715:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff920a48351c00 x1875444570029056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575960.028738:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570029056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575960.028748:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575960.028751:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575960.028755:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570029056:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575960.028826:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.028830:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575960.028832:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.028837:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12713 00000100:00080000:0.0:1788575960.028842:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.028845:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575960.028851:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575960.028864:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570029312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575960.028933:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575960.028961:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575960.028975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff920a48351500 x1875444570029184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575960.028994:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570029184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575960.029003:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575960.029006:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575960.029010:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029184:192.168.202.157@tcp:38:kworker.0 00000100:00080000:0.0:1788575960.029059:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.029073:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575960.029075:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.029080:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12713 00000100:00080000:0.0:1788575960.029084:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.029088:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575960.029094:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575960.029109:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570029440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575960.029118:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575960.029121:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575960.029123:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575960.029127:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12713 00000100:00080000:0.0:1788575960.029130:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575960.029134:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575960.029138:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575960.029148:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570029568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575960.029256:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575960.029282:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575960.029294:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff920a48350a80 x1875444570029312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575960.029306:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570029312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575960.029320:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575960.029323:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575960.029326:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575960.029332:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575960.029351:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575960.029361:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff920a48351500 x1875444570029440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575960.029371:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570029440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575960.029378:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575960.029381:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575960.029384:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575960.029389:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575960.029406:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575960.029416:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575960/real 1788575960] req@ffff920a48351c00 x1875444570029568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575960.029427:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570029568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575960.029434:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575960.029437:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575960.029440:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575960.475746:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575961.051803:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575961.499766:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575962.075715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575962.524766:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575963.099812:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575963.547874:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575964.123682:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575964.571862:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788575965.147920:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575965.147929:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788575965.147933:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575965.147944:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12718 00000100:00080000:1.0:1788575965.147954:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575965.147961:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575965.148001:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:1.0:1788575965.148001:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444570029696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575965.148041:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575965.148045:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788575965.148047:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575965.148051:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12718 00000100:00080000:1.0:1788575965.148056:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575965.148061:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575965.148071:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575965.148084:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444570029824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575965.148094:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575965.148098:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788575965.148099:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575965.148104:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12718 00000100:00080000:1.0:1788575965.148107:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575965.148111:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575965.148117:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788575965.148129:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570029952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575965.148140:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575965.148143:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788575965.148145:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575965.148150:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12718 00000100:00080000:1.0:1788575965.148154:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575965.148157:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575965.148163:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575965.148175:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570030080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788575965.148184:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788575965.148187:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788575965.148188:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788575965.148192:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12718 00000100:00080000:1.0:1788575965.148196:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788575965.148200:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788575965.148205:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788575965.148215:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570030208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575965.150931:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570029696:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788575965.150997:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575965.151031:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff920a52890700 x1875444570029696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575965.151048:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444570029696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575965.151058:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575965.151062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575965.151066:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570029696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575965.151073:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575965.151093:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575965.151103:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff920a52893100 x1875444570029824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575965.151115:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444570029824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575965.151123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575965.151126:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575965.151129:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575965.151138:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575965.151154:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575965.151162:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff920a52891180 x1875444570029952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575965.151173:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570029952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575965.151181:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575965.151184:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575965.151187:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570029952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575965.151193:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575965.151208:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575965.151216:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff920a52892300 x1875444570030080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575965.151226:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570030080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575965.151233:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575965.151236:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575965.151239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575965.151244:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575965.151258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575965.151266:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575965/real 1788575965] req@ffff920a52892d80 x1875444570030208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575965.151277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570030208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575965.151283:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575965.151286:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575965.151289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575965.595759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575966.171738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575966.619765:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575967.195753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575967.643829:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575968.219712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575968.667768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575969.243726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575969.691771:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575970.267769:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575970.267787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575970.267789:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575970.267795:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.267807:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12723 00000100:00080000:0.0:1788575970.267818:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.267825:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575970.267863:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570030336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.267895:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.267899:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575970.267901:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.267904:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12723 00000100:00080000:0.0:1788575970.267907:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.267911:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575970.267919:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575970.267929:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570030464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.267940:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.267944:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575970.267946:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.267950:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12723 00000100:00080000:0.0:1788575970.267954:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.267957:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575970.267961:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575970.267969:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570030592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.267981:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.267987:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575970.267989:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.267996:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12723 00000100:00100000:2.0:1788575970.267997:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570030336:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575970.268000:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.268012:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575970.268019:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788575970.268093:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575970.268136:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff920a48353b80 x1875444570030336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575970.268156:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570030336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.268168:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.268173:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575970.268178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570030336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575970.268186:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575970.268210:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575970.268223:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff920a48353480 x1875444570030464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575970.268229:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570030720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575970.268249:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575970.268252:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575970.268253:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575970.268257:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12723 00000100:00080000:0.0:1788575970.268262:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575970.268266:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575970.268271:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575970.268283:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570030848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575970.268749:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570030464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.268759:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.268764:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575970.268767:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575970.268813:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575970.268838:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575970.268849:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff920a48353800 x1875444570030592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575970.268861:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570030592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.268869:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.268872:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575970.268875:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575970.268881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575970.268898:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575970.268907:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff920a48350700 x1875444570030720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575970.268918:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570030720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.268926:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.268929:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575970.268932:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575970.268938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575970.268953:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575970.268962:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575970/real 1788575970] req@ffff920a48350380 x1875444570030848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575970.268972:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570030848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788575995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575970.268980:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575970.268983:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575970.268986:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570030848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575970.717370:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575971.291876:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575971.739915:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575972.316668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575972.763855:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575973.339725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575973.787811:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575974.363830:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575974.811767:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575975.388287:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575975.388916:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.388927:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575975.388932:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.388946:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12728 00000100:00080000:0.0:1788575975.388958:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.388966:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575975.389022:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570030976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.389055:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.389059:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575975.389062:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.389066:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12728 00000100:00080000:0.0:1788575975.389070:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.389076:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575975.389090:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575975.389105:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570030976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575975.389109:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570031104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.389126:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.389130:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575975.389133:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.389139:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12728 00000100:00080000:0.0:1788575975.389144:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.389149:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575975.389156:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575975.389170:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570031232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.389180:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.389184:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575975.389186:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575975.389190:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12728 00000100:00080000:0.0:1788575975.389194:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.389198:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575975.389204:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575975.389215:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570031360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575975.389224:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575975.389227:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575975.389229:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1788575975.389238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575975.389243:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12728 00000100:00080000:0.0:1788575975.389249:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575975.389255:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575975.389262:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575975.389284:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570031488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575975.389285:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff920a48350380 x1875444570030976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575975.389362:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570030976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.389372:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.389375:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575975.389378:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570030976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575975.389471:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575975.389492:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575975.389501:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff920a48350700 x1875444570031104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575975.389508:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570031104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.389514:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.389516:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575975.389518:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575975.389521:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575975.389531:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575975.389537:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff920a48353800 x1875444570031232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575975.389543:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570031232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.389556:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.389558:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575975.389560:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575975.389564:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575975.389576:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575975.389582:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff920a48353480 x1875444570031360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575975.389611:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570031360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.389616:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.389617:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575975.389619:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575975.389622:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575975.389634:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575975.389640:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575975/real 1788575975] req@ffff920a48353b80 x1875444570031488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575975.389646:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570031488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575975.389650:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575975.389652:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575975.389653:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575975.835806:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575976.411696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575976.859813:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575977.435745:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575977.883744:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575978.462091:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575978.907799:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788575979.483700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575979.931752:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575980.507690:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788575980.507695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575980.507699:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575980.507704:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575980.507713:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12733 00000100:00080000:0.0:1788575980.507722:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575980.507729:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575980.507759:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570031616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575980.507782:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575980.507786:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575980.507787:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575980.507791:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12733 00000100:00080000:0.0:1788575980.507794:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575980.507797:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575980.507806:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575980.507819:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570031744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575980.507828:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575980.507830:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575980.507832:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788575980.507833:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570031616:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575980.507835:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12733 00000100:00080000:0.0:1788575980.507839:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575980.507842:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575980.507846:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575980.507854:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570031872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575980.507861:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575980.507864:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575980.507865:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575980.507868:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12733 00000100:00080000:0.0:1788575980.507870:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575980.507873:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575980.507876:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575980.507884:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570032000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575980.507889:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575980.507892:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575980.507895:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575980.507896:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575980.507899:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12733 00000100:00080000:0.0:1788575980.507901:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575980.507904:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575980.507930:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575980.507931:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff920a48351c00 x1875444570031616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575980.507939:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570032128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575980.507946:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570031616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575980.507957:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575980.507960:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575980.507964:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570031616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575980.507985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575980.508002:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575980.508010:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff920a48351500 x1875444570031744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575980.508017:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570031744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575980.508022:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575980.508024:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575980.508026:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575980.508029:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575980.508039:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575980.508044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff920a48350a80 x1875444570031872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575980.508050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570031872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575980.508054:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575980.508056:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575980.508057:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570031872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575980.508060:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575980.508070:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575980.508075:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff920a48353b80 x1875444570032000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575980.508080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570032000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575980.508084:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575980.508086:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575980.508087:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575980.508090:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575980.508102:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575980.508111:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575980/real 1788575980] req@ffff920a48353480 x1875444570032128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575980.508121:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570032128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575980.508127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575980.508130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575980.508133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575980.955729:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575981.531686:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575981.980821:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575982.555766:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575983.003840:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575983.579754:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575984.027796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575984.603715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575985.051784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788575985.627731:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788575985.627732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575985.627743:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575985.627748:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575985.627761:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12739 00000100:00080000:0.0:1788575985.627774:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575985.627785:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575985.627839:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570032256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575985.627859:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575985.627862:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575985.627863:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575985.627866:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12739 00000100:00080000:0.0:1788575985.627870:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575985.627874:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575985.627886:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575985.627896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570032256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575985.627902:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570032384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575985.627914:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575985.627917:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575985.627918:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575985.627922:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12739 00000100:00080000:0.0:1788575985.627925:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575985.627927:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575985.627932:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575985.627941:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570032512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575985.627948:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575985.627950:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575985.627952:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575985.627956:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12739 00000100:00080000:0.0:1788575985.627959:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575985.627962:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:2.0:1788575985.627968:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:0.0:1788575985.627969:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788575985.627985:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570032640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575985.627995:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575985.627998:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575985.628000:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575985.628004:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12739 00000100:00080000:0.0:1788575985.628007:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575985.628012:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575985.628017:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575985.628022:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575985/real 1788575985] req@ffff920a48353800 x1875444570032256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575985.628027:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570032768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575985.628058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570032256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575985.628082:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575985.628087:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575985.628091:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570032256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575985.628119:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575985.628141:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575985.628154:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575985/real 1788575985] req@ffff920a48350700 x1875444570032384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575985.628165:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570032384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575985.628172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575985.628175:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575985.628178:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575985.628184:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575985.628203:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575985.628213:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575985/real 1788575985] req@ffff920a48350380 x1875444570032512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575985.628231:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570032512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575985.628239:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575985.628242:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575985.628246:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575985.628251:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575985.628273:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575985.628283:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575985/real 1788575985] req@ffff920a48353480 x1875444570032640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575985.628294:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570032640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575985.628301:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575985.628304:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575985.628307:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575985.628313:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575985.628332:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575985.628341:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575985/real 1788575985] req@ffff920a48353b80 x1875444570032768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575985.628360:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570032768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575985.628373:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575985.628383:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575985.628386:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570032768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575986.075783:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575986.651748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575987.099741:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575987.675696:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575988.123764:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575988.699717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575989.147769:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575989.723660:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575990.171755:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575990.747690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788575990.747731:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575990.747736:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788575990.747739:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575990.747749:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12744 00000100:00080000:0.0:1788575990.747759:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575990.747766:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575990.747806:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570032896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575990.747832:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575990.747835:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788575990.747837:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575990.747840:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12744 00000100:00080000:0.0:1788575990.747844:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575990.747848:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575990.747856:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575990.747873:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570033024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575990.747882:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570032896:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788575990.747883:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575990.747891:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788575990.747893:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575990.747898:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12744 00000100:00080000:0.0:1788575990.747902:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575990.747906:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575990.747912:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575990.747935:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570033152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788575990.747946:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788575990.747948:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788575990.747951:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788575990.747953:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788575990.747957:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12744 00000100:00080000:0.0:1788575990.747961:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575990.747965:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575990.747971:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575990.747980:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575990/real 1788575990] req@ffff920a48350a80 x1875444570032896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575990.747986:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570033280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788575990.748002:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788575990.748005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570032896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575990.748007:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788575990.748012:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575990.748018:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575990.748019:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12744 00000100:00080000:0.0:1788575990.748024:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788575990.748028:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788575990.748034:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788575990.748034:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575990.748043:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570032896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788575990.748049:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570033408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788575990.748068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575990.748092:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575990.748102:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575990/real 1788575990] req@ffff920a48351500 x1875444570033024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575990.748110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570033024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575990.748115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575990.748118:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575990.748120:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033024:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575990.748124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788575990.748136:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575990.748141:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575990/real 1788575990] req@ffff920a48351c00 x1875444570033152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575990.748148:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570033152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575990.748153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575990.748156:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575990.748159:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788575990.748164:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575990.748183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575990.748192:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575990/real 1788575990] req@ffff920a48353b80 x1875444570033280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575990.748204:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570033280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575990.748210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575990.748212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575990.748214:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788575990.748231:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788575990.748245:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788575990.748252:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575990/real 1788575990] req@ffff920a48353480 x1875444570033408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788575990.748259:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570033408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575990.748263:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788575990.748265:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788575990.748267:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788575991.195774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575991.771688:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575992.219795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575992.795708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575993.243771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575993.819698:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575994.267788:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575994.843706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575995.291757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788575995.867683:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788575995.867684:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788575995.867694:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788575995.867698:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575995.867708:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12749 00000100:00080000:2.0:1788575995.867715:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575995.867721:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788575995.867748:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570033536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575995.867769:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575995.867772:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788575995.867773:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575995.867776:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12749 00000100:00080000:2.0:1788575995.867778:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575995.867782:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575995.867790:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788575995.867796:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570033536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575995.867803:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570033664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575995.867811:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575995.867814:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788575995.867815:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575995.867818:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12749 00000100:00080000:2.0:1788575995.867820:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575995.867822:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575995.867825:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788575995.867833:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570033792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575995.867839:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575995.867841:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788575995.867843:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788575995.867845:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12749 00000100:00080000:2.0:1788575995.867846:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788575995.867850:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788575995.867851:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788575995.867855:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788575995.867862:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570033920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788575995.867871:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788575995.867875:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788575995.867877:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788575995.867879:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575995/real 1788575995] req@ffff920a60ac4000 x1875444570033536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788575995.867883:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12749 00000100:00080000:2.0:1788575995.867891:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788575995.867897:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788575995.867899:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570033536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1788575995.867904:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:0.0:1788575995.867908:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575995.867912:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575995.867914:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570033536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788575995.867920:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444570034048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788575995.867932:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575995.867948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575995.867956:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575995/real 1788575995] req@ffff920a60ac4700 x1875444570033664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575995.867963:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570033664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575995.867968:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575995.867970:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575995.867971:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575995.867974:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788575995.867986:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575995.867999:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575995/real 1788575995] req@ffff920a60ac4380 x1875444570033792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575995.868006:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570033792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575995.868010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575995.868012:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575995.868014:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788575995.868017:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575995.868027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575995.868032:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575995/real 1788575995] req@ffff920a60ac5c00 x1875444570033920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575995.868038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570033920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575995.868043:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575995.868044:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575995.868046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570033920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788575995.868066:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788575995.868084:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788575995.868094:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788575995/real 1788575995] req@ffff920a60ac7800 x1875444570034048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788575995.868110:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444570034048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788575995.868115:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788575995.868117:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788575995.868119:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788575996.315724:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788575996.891675:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575997.339721:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575997.915755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788575998.363785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575998.939695:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788575999.388199:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788575999.963709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576000.411825:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576000.987737:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788576000.987746:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576000.987752:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576000.987758:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576000.987771:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12754 00000100:00080000:2.0:1788576000.987785:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576000.987793:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576000.987845:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444570034176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576000.987882:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576000.987886:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576000.987888:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576000.987892:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12754 00000100:00080000:2.0:1788576000.987895:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576000.987900:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576000.987913:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576000.987944:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570034176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576000.987946:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444570034304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576000.987968:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576000.987971:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576000.987973:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576000.987979:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12754 00000100:00080000:2.0:1788576000.987983:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576000.987987:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576000.987993:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576000.988005:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570034432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576000.988014:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576000.988017:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576000.988019:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1788576000.988025:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788576000.988026:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12754 00000100:00080000:2.0:1788576000.988035:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576000.988040:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576000.988046:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576000.988059:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570034560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576000.988068:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576000.988071:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576000.988073:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576000.988077:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12754 00000100:00080000:2.0:1788576000.988081:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576000.988083:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576000/real 1788576000] req@ffff920a60ac4e00 x1875444570034176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576000.988089:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576000.988097:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576000.988115:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570034688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576000.988122:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444570034176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576000.988153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576000.988158:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576000.988163:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570034176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576000.988190:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576000.988217:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576000.988230:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576000/real 1788576000] req@ffff920a60ac7800 x1875444570034304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576000.988243:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444570034304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576000.988257:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576000.988260:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576000.988264:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576000.988278:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576000.988298:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576000.988310:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576000/real 1788576000] req@ffff920a60ac5c00 x1875444570034432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576000.988322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570034432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576000.988330:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576000.988333:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576000.988336:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576000.988342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576000.988361:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576000.988371:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576000/real 1788576000] req@ffff920a60ac4380 x1875444570034560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576000.988381:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570034560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576000.988389:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576000.988392:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576000.988396:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576000.988402:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576000.988419:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576000.988428:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576000/real 1788576000] req@ffff920a60ac4700 x1875444570034688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576000.988437:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570034688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576025 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576000.988445:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576000.988448:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576000.988451:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576001.435806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576002.011767:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576002.459855:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576003.035757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576003.483807:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576004.059731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576004.507800:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576005.083726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576005.533474:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576006.107742:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1788576006.107744:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576006.107754:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576006.107759:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.107771:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12759 00000100:00080000:0.0:1788576006.107783:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.107792:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576006.107837:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570034816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576006.107873:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.107876:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576006.107878:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.107883:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12759 00000100:00080000:0.0:1788576006.107886:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.107890:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576006.107902:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576006.107913:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570034944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576006.107922:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.107926:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576006.107928:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.107932:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12759 00000100:00080000:0.0:1788576006.107936:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.107939:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576006.107943:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576006.107955:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570035072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576006.107955:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570034816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788576006.107970:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.107974:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576006.107975:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.107978:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12759 00000100:00080000:0.0:1788576006.107981:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.107984:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576006.107989:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576006.108001:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570035200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576006.108017:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576006.108021:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576006.108023:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576006.108027:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12759 00000100:00080000:0.0:1788576006.108030:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576006.108034:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576006.108039:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576006.108049:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570035328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576006.108068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576006.108123:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576006.108145:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576006.108182:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576006.555832:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576007.131747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576007.579752:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576008.155719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576008.603866:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576009.179747:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:2.0:1788576009.180631:0:2385:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788576009.180679:0:2385:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788576009.180696:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576009.180747:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576009.180757:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576009.180766:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576009.180783:0:2385:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788576009.180975:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576009] req@ffff920a48353800 x1875444570035072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576009.180999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570035072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576009.181010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576009.181014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576009.181018:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576009.181026:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576009] req@ffff920a48353480 x1875444570035200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576009.181038:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570035200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576009.181045:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576009.181048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576009.181051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576009.181058:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576009] req@ffff920a48353b80 x1875444570035328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576009.181068:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570035328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576009.181075:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576009.181079:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576009.181082:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035328:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576009.181088:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576009] req@ffff920a48350380 x1875444570034816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576009.181101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570034816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576009.181108:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576009.181111:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576009.181114:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570034816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576009.181120:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576006/real 1788576009] req@ffff920a48350700 x1875444570034944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576009.181131:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570034944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576009.181139:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576009.181143:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576009.181146:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570034944:192.168.202.157@tcp:38:kworker.0 00010000:00010000:2.0:1788576009.631031:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576010.203784:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576010.651795:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576011.227738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576011.227753:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576011.227763:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576011.227769:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576011.227782:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12764 00000100:00080000:0.0:1788576011.227796:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576011.227806:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576011.227892:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570035456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576011.227928:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576011.227932:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576011.227934:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576011.227938:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12764 00000100:00080000:0.0:1788576011.227942:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576011.227946:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576011.227958:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576011.227972:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570035584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576011.227983:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576011.227988:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576011.227993:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788576011.227995:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570035456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788576011.228000:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12764 00000100:00080000:0.0:1788576011.228013:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576011.228019:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576011.228024:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576011.228036:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570035712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576011.228045:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576011.228049:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576011.228051:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576011.228057:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12764 00000100:00080000:0.0:1788576011.228061:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576011.228066:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576011.228070:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576011.228086:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570035840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788576011.228087:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:0.0:1788576011.228101:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576011.228105:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576011.228108:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576011.228125:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12764 00000100:00080000:0.0:1788576011.228131:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788576011.228137:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff920a48350700 x1875444570035456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576011.228138:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576011.228147:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576011.228164:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570035968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576011.228174:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570035456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576011.228224:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576011.228229:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576011.228236:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570035456:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576011.228361:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035584:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576011.228401:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576011.228416:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff920a48350380 x1875444570035584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576011.228428:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570035584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576011.228436:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576011.228440:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576011.230095:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576011.230124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035712:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576011.231500:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576011.231537:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff920a48353b80 x1875444570035712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576011.231558:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570035712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576011.231569:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576011.231573:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576011.231576:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576011.231601:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035840:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576011.231630:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576011.231642:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff920a48353480 x1875444570035840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576011.231654:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570035840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576011.231662:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576011.231665:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576011.231668:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576011.231674:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035968:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576011.231693:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576011.231704:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576011/real 1788576011] req@ffff920a48353800 x1875444570035968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576011.231714:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570035968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576011.231727:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576011.231730:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576011.231733:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570035968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576011.677689:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576012.251707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576012.699924:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576013.275690:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576013.723789:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576014.300027:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576014.747849:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576015.323678:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576015.771790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576016.347709:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576016.348716:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.348722:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576016.348726:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.348736:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12769 00000100:00080000:3.0:1788576016.348746:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.348754:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576016.348792:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570036096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.348821:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.348825:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576016.348827:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.348831:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12769 00000100:00080000:3.0:1788576016.348835:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.348840:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576016.348849:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576016.348862:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570036224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.348871:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788576016.348872:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570036096:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576016.348874:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576016.348876:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.348880:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12769 00000100:00080000:3.0:1788576016.348883:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.348886:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576016.348891:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576016.348899:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570036352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.348907:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.348910:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576016.348912:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.348916:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12769 00000100:00080000:3.0:1788576016.348928:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.348932:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576016.348936:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788576016.348938:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576016.348945:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570036480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576016.348953:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576016.348956:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576016.348957:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576016.348960:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12769 00000100:00080000:3.0:1788576016.348963:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576016.348966:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576016.348969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff920a529b1180 x1875444570036096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1788576016.348970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576016.348980:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570036608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576016.348984:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570036096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.349010:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576016.349014:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576016.349017:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570036096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576016.349049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576016.349072:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576016.349084:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff920a529b1500 x1875444570036224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576016.349096:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570036224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.349110:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576016.349113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576016.349117:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576016.349122:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576016.349148:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576016.349158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff920a529b2300 x1875444570036352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576016.349168:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570036352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.349175:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576016.349186:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576016.349190:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576016.349195:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576016.349213:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576016.349222:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff920a529b0e00 x1875444570036480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576016.349233:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570036480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.349240:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576016.349243:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576016.349247:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576016.349252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576016.349268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576016.349278:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576016/real 1788576016] req@ffff920a529b3b80 x1875444570036608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576016.349288:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570036608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576016.349295:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576016.349298:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576016.349301:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576016.795772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576017.371705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576017.819768:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576018.395712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576018.844127:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576019.419769:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576019.867829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576020.443916:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576020.891772:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576021.467728:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576021.468700:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576021.468709:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576021.468714:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576021.468725:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12774 00000100:00080000:1.0:1788576021.468736:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576021.468742:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576021.468779:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570036736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576021.468808:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576021.468812:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576021.468814:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576021.468818:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12774 00000100:00080000:1.0:1788576021.468822:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576021.468826:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576021.468836:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576021.468849:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570036864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576021.468859:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576021.468864:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00100000:2.0:1788576021.468864:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570036736:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788576021.468867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576021.468876:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12774 00000100:00080000:1.0:1788576021.468880:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576021.468884:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576021.468889:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576021.468901:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570036992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576021.468916:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576021.468920:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576021.468921:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576021.468925:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12774 00000100:00080000:1.0:1788576021.468929:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788576021.468933:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788576021.468937:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576021.468943:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576021.468969:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff920a52893800 x1875444570036736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576021.468986:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570036736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576021.468996:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576021.469000:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576021.469003:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570036736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576021.469025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576021.469047:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576021.469059:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff920a52890e00 x1875444570036864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576021.469070:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570036864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576021.469078:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576021.469081:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576021.469084:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576021.469089:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576021.469107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576021.469119:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff920a52891500 x1875444570036992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576021.469129:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570036992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576021.469136:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576021.469139:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576021.469142:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570036992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788576021.469238:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570037120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576021.469258:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576021.469261:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576021.469262:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576021.469266:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12774 00000100:00080000:1.0:1788576021.469270:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576021.469274:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576021.469279:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576021.469290:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570037248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576021.469349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576021.469385:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576021.469402:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff920a52893480 x1875444570037120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576021.469420:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570037120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576021.469489:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576021.469492:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576021.469496:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576021.469502:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576021.469523:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576021.469535:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576021/real 1788576021] req@ffff920a52891880 x1875444570037248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576021.469546:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570037248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576021.469554:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576021.469557:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576021.469560:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576021.917346:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576022.491699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576022.940509:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576023.515943:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576023.964086:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576024.539775:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576024.988314:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576025.563714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576026.015432:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576026.587732:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.587744:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576026.587749:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.587759:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12780 00000100:00080000:0.0:1788576026.587770:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.587777:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576026.587779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788576026.587817:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570037376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.587847:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.587852:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576026.587853:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.587857:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12780 00000100:00080000:0.0:1788576026.587861:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.587865:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576026.587874:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576026.587885:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570037504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.587894:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.587898:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576026.587900:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.587903:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12780 00000100:00080000:0.0:1788576026.587907:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.587910:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576026.587914:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576026.587932:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570037632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.587941:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.587945:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576026.587946:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.587950:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12780 00000100:00080000:0.0:1788576026.587954:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.587958:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576026.587963:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576026.587967:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570037376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576026.587975:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570037760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576026.587985:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576026.587988:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576026.587990:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576026.587993:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12780 00000100:00080000:0.0:1788576026.587996:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576026.588000:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576026.588004:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576026.588013:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570037888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1788576026.588032:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576026.588069:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff920a48351c00 x1875444570037376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576026.588108:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570037376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576026.588118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576026.588123:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576026.588128:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570037376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576026.588137:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576026.588161:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576026.588174:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff920a48351500 x1875444570037504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576026.588185:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570037504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576026.588193:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576026.588197:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576026.588200:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576026.588206:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576026.588230:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576026.588244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff920a48350a80 x1875444570037632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576026.588257:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570037632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576026.588270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576026.588273:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576026.588276:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037632:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576026.588325:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576026.588347:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576026.588358:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff920a48353800 x1875444570037760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576026.588368:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570037760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576026.588375:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576026.588378:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576026.588381:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576026.588386:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576026.588402:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576026.588413:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576026/real 1788576026] req@ffff920a48353480 x1875444570037888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576026.588423:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570037888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576026.588429:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576026.588432:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576026.588434:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570037888:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576027.035902:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576027.616062:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576028.059814:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576028.635982:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576029.083809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576029.662030:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576030.107763:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576030.683703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576031.131820:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576031.707805:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576031.707844:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576031.707848:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576031.707851:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576031.707858:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12785 00000100:00080000:1.0:1788576031.707872:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576031.707880:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576031.707910:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570038016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576031.707942:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576031.707946:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576031.707948:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576031.707952:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12785 00000100:00080000:1.0:1788576031.707956:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576031.707960:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576031.707970:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576031.707980:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570038144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576031.707986:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576031.707989:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576031.707990:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576031.707993:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12785 00000100:00080000:1.0:1788576031.707995:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576031.707997:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576031.707999:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576031.708005:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570038272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576031.708010:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576031.708012:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576031.708013:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576031.708015:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12785 00000100:00080000:1.0:1788576031.708017:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576031.708018:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576031.708021:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576031.708025:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570038400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576031.708030:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576031.708031:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576031.708032:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576031.708034:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12785 00000100:00080000:1.0:1788576031.708035:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576031.708037:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576031.708039:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576031.708044:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570038528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576031.708094:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570038016:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576031.708147:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576031.708178:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576031/real 1788576031] req@ffff920a52891f80 x1875444570038016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576031.708194:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570038016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576031.708204:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576031.708207:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576031.708210:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570038016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576031.708215:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576031.708243:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576031.708255:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576031/real 1788576031] req@ffff920a52890380 x1875444570038144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576031.708279:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570038144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576031.708285:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576031.708287:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576031.708289:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576031.708292:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576031.708307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576031.708313:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576031/real 1788576031] req@ffff920a52893b80 x1875444570038272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576031.708320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570038272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576031.708324:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576031.708326:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576031.708328:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576031.708330:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576031.708340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576031.708345:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576031/real 1788576031] req@ffff920a52890000 x1875444570038400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576031.708360:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570038400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576031.708367:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576031.708370:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576031.708372:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576031.708377:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576031.708395:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576031.708405:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576031/real 1788576031] req@ffff920a52892680 x1875444570038528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576031.708419:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570038528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576031.708425:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576031.708427:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576031.708428:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576032.156078:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576032.731779:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576033.179804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576033.755718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576034.203851:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576034.779794:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576035.227804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576035.803708:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576036.251829:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1788576036.827701:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576036.827712:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1788576036.827714:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576036.827718:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576036.827730:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12790 00000100:00080000:1.0:1788576036.827742:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576036.827750:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576036.827787:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890a80 x1875444570038656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576036.827821:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576036.827826:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576036.827828:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576036.827832:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12790 00000100:00080000:1.0:1788576036.827836:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576036.827839:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576036.827848:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576036.827861:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892a00 x1875444570038784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576036.827872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576036.827875:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576036.827877:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576036.827881:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12790 00000100:00080000:1.0:1788576036.827884:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576036.828685:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576036.828687:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570038656:192.168.202.157@tcp:250:kworker.0 00000008:00100000:1.0:1788576036.828704:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576036.828727:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570038912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576036.828746:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576036.828751:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1788576036.828754:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:1.0:1788576036.828755:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576036.828762:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12790 00000100:00080000:1.0:1788576036.828769:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576036.828773:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576036.828779:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576036.828793:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570039040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576036.828796:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576036/real 1788576036] req@ffff920a52890a80 x1875444570038656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576036.828815:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576036.828820:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576036.828822:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1788576036.828827:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890a80 x1875444570038656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576036.828828:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12790 00000100:00080000:1.0:1788576036.828835:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576036.828845:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00080000:2.0:1788576036.828848:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1788576036.828850:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00080000:2.0:1788576036.828857:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576036.828863:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570038656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788576036.828867:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570039168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576036.828874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576036.828902:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576036.828914:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576036/real 1788576036] req@ffff920a52892a00 x1875444570038784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576036.828933:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892a00 x1875444570038784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576036.828942:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576036.828945:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576036.828948:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576036.828968:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576036.828984:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576036.828994:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576036/real 1788576036] req@ffff920a52892680 x1875444570038912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576036.829005:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570038912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576036.829013:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576036.829016:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576036.829020:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570038912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576036.829025:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576036.829040:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576036.829048:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576036/real 1788576036] req@ffff920a52890000 x1875444570039040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576036.829058:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570039040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576036.829072:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576036.829075:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576036.829078:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576036.829084:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576036.829100:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576036.829108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576036/real 1788576036] req@ffff920a52893b80 x1875444570039168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576036.829119:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570039168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576036.829127:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576036.829130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576036.829133:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576037.275804:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576037.851729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576038.299946:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576038.875794:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576039.323829:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576039.899755:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576040.347798:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576040.923717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576041.372615:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576041.947727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576041.947925:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576041.947931:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576041.947935:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576041.947944:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12795 00000100:00080000:0.0:1788576041.947954:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576041.947962:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576041.947997:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570039296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576041.948037:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576041.948040:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576041.948042:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576041.948045:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12795 00000100:00080000:0.0:1788576041.948048:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576041.948051:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576041.948059:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576041.948069:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570039424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576041.948077:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576041.948080:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576041.948082:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576041.948085:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12795 00000100:00080000:0.0:1788576041.948088:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576041.948092:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576041.948095:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576041.948105:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570039552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576041.948113:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576041.948116:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576041.948117:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576041.948121:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12795 00000100:00080000:0.0:1788576041.948129:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576041.948135:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576041.948139:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576041.948148:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570039680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576041.948153:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576041.948156:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576041.948157:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576041.948159:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12795 00000100:00080000:0.0:1788576041.948162:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576041.948165:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576041.948169:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576041.948178:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570039808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576041.948260:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570039296:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576041.948314:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576041.948347:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576041/real 1788576041] req@ffff920a48353b80 x1875444570039296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576041.948363:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570039296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576041.948373:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576041.948376:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576041.948379:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570039296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576041.948384:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576041.948400:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576041.948408:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576041/real 1788576041] req@ffff920a48350380 x1875444570039424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576041.948416:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570039424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576041.948421:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576041.948423:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576041.948425:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576041.948428:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576041.948437:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576041.948442:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576041/real 1788576041] req@ffff920a48350700 x1875444570039552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576041.948448:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570039552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576041.948453:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576041.948454:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576041.948456:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576041.948458:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576041.948469:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576041.948474:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576041/real 1788576041] req@ffff920a48353480 x1875444570039680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576041.948479:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570039680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576041.948484:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576041.948486:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576041.948488:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576041.948491:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576041.948499:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576041.948504:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576041/real 1788576041] req@ffff920a48353800 x1875444570039808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576041.948510:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570039808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576066 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576041.948515:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576041.948516:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576041.948518:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570039808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576042.395735:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576042.971722:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576043.419777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576043.995717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576044.443809:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576045.019700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576045.467821:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576046.043787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576046.491779:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576047.067798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576047.068273:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576047.068279:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576047.068284:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576047.068294:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12800 00000100:00080000:2.0:1788576047.068304:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576047.068312:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576047.068348:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570039936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576047.068379:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576047.068383:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576047.068385:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576047.068389:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12800 00000100:00080000:2.0:1788576047.068392:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576047.068396:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576047.068405:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576047.068416:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570040064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576047.068424:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576047.068427:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576047.068429:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576047.068433:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12800 00000100:00080000:2.0:1788576047.068436:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576047.068439:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576047.068442:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576047.068451:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570040192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576047.068464:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576047.068467:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576047.068468:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576047.068471:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12800 00000100:00080000:2.0:1788576047.068474:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576047.068477:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576047.068481:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576047.068490:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570040320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576047.068497:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576047.068501:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576047.068502:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576047.068505:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12800 00000100:00080000:2.0:1788576047.068508:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576047.068511:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576047.068515:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576047.068523:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570040448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576047.068610:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570039936:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576047.068675:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576047.068720:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff920a60ac4000 x1875444570039936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.068736:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570039936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576047.068745:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576047.068749:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576047.068753:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570039936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576047.068759:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576047.068781:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576047.068793:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff920a60ac6a00 x1875444570040064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.068805:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570040064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576047.068812:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576047.068815:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576047.068818:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040064:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576047.068823:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576047.068839:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576047.068848:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff920a60ac7100 x1875444570040192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.068858:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570040192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576047.068865:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576047.068868:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576047.068870:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576047.068875:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576047.068892:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576047.068902:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff920a60ac6300 x1875444570040320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.068911:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570040320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576047.068918:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576047.068921:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576047.068924:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576047.068930:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576047.068947:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576047.068956:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576047/real 1788576047] req@ffff920a60ac5180 x1875444570040448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576047.068967:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570040448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576047.068974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576047.068977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576047.068981:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576047.515785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576048.091738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576048.539777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576049.115677:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576049.563751:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576050.139725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576050.590804:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576051.163700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576051.611834:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576052.187718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576052.187763:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576052.187767:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576052.187771:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576052.187781:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12805 00000100:00080000:0.0:1788576052.187791:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576052.187797:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576052.187832:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570040576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576052.187853:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576052.187856:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576052.187857:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576052.187860:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12805 00000100:00080000:0.0:1788576052.187864:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576052.187867:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576052.187875:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576052.187881:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570040576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576052.187885:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570040704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576052.187893:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576052.187896:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576052.187897:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576052.187899:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12805 00000100:00080000:0.0:1788576052.187901:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576052.187903:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576052.187905:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576052.187914:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570040832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576052.187922:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576052.187925:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576052.187926:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576052.187930:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12805 00000100:00080000:0.0:1788576052.187934:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576052.187938:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576052.187944:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:2.0:1788576052.187956:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576052.187959:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570040960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576052.187968:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576052.187970:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576052.187971:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576052.187975:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12805 00000100:00080000:0.0:1788576052.187979:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576052.187981:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576052.187985:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576052.187992:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570041088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576052.187998:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff920a48350a80 x1875444570040576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.188016:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570040576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.188023:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.188027:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576052.188030:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570040576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576052.188049:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576052.188066:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576052.188074:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff920a48351500 x1875444570040704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.188081:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570040704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.188086:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.188088:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576052.188090:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576052.188093:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576052.188103:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576052.188108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff920a48351c00 x1875444570040832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.188114:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570040832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.188122:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.188124:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576052.188126:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576052.188128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576052.188139:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576052.188143:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff920a48353800 x1875444570040960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.188149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570040960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.188153:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.188155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576052.188156:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570040960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576052.188159:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576052.188168:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576052.188173:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576052/real 1788576052] req@ffff920a48353480 x1875444570041088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576052.188178:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570041088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576052.188184:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576052.188187:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576052.188191:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041088:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576052.635859:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576053.211663:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576053.659809:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576054.235813:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576054.683832:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576055.260307:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576055.707837:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576056.283815:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576056.731790:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576057.307694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576057.307707:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576057.307713:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576057.307717:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576057.307728:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12810 00000100:00080000:2.0:1788576057.307749:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576057.307757:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576057.307795:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570041216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576057.307821:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576057.307824:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576057.307826:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576057.307831:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12810 00000100:00080000:2.0:1788576057.307835:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576057.307841:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576057.307850:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576057.307856:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570041216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576057.307863:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444570041344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576057.307870:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576057.307873:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576057.307875:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576057.307878:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12810 00000100:00080000:2.0:1788576057.307881:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576057.307883:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576057.307885:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576057.307895:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570041472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576057.307903:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576057.307906:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576057.307908:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576057.307912:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12810 00000100:00080000:2.0:1788576057.307916:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576057.307920:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576057.307925:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576057.307938:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570041600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576057.307939:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788576057.307948:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576057.307951:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576057.307952:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576057.307955:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12810 00000100:00080000:2.0:1788576057.307959:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576057.307961:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576057.307964:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576057.307972:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570041728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576057.307981:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576057] req@ffff920a60ac4a80 x1875444570041216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576057.307999:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570041216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576057.308014:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576057.308018:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576057.308021:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570041216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576057.308042:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576057.308065:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576057.308077:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576057] req@ffff920a60ac5500 x1875444570041344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576057.308089:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444570041344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576057.308098:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576057.308101:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576057.308104:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576057.308110:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576057.308128:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576057.308138:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576057] req@ffff920a60ac5180 x1875444570041472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576057.308149:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570041472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576057.308156:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576057.308159:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576057.308162:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576057.308168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576057.308189:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576057.308200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576057] req@ffff920a60ac6300 x1875444570041600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576057.308210:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570041600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576057.308218:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576057.308221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576057.308224:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041600:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576057.308230:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576057.308248:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576057.308269:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576057/real 1788576057] req@ffff920a60ac7100 x1875444570041728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576057.308281:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570041728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576057.308288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576057.308291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576057.308294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576057.755790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576058.331762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576058.779792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576059.355711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576059.803780:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576060.379706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576060.827791:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576061.403703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576061.851930:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576062.427744:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1788576062.427758:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576062.427774:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576062.427779:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.427793:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12815 00000100:00080000:2.0:1788576062.427804:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.427814:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576062.427853:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570041856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.427883:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.427887:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576062.427889:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.427898:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12815 00000100:00080000:2.0:1788576062.427902:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.427906:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576062.427917:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576062.427932:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570041984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.427942:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.427946:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576062.427948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.427954:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12815 00000100:00080000:2.0:1788576062.427958:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.427962:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576062.427968:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576062.427981:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570042112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.427991:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.427994:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576062.427995:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.427998:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12815 00000100:00080000:2.0:1788576062.428000:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.428003:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576062.428007:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576062.428015:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570042240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576062.428022:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576062.428024:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576062.428025:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576062.428028:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12815 00000100:00080000:2.0:1788576062.428030:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576062.428032:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576062.428036:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576062.428042:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570041856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576062.428048:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570042368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1788576062.428119:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576062.428221:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff920a60ac6a00 x1875444570041856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.428240:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570041856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.428251:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.428255:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576062.428259:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570041856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788576062.428267:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576062.428449:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576062.428524:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff920a60ac4000 x1875444570041984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.428547:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570041984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.428557:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.428560:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576062.428564:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570041984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576062.428571:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576062.428641:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576062.428653:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff920a60ac4700 x1875444570042112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.428671:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570042112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.428679:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.428682:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576062.428685:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576062.428691:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576062.428711:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576062.428719:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff920a60ac4380 x1875444570042240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.428726:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570042240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.428731:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.428733:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576062.428735:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788576062.428772:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576062.428786:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576062.428792:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576062/real 1788576062] req@ffff920a60ac5c00 x1875444570042368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576062.428799:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570042368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576062.428808:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576062.428810:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576062.428812:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576062.875781:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576063.451845:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576063.899818:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576064.475738:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576064.924732:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576065.499718:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576065.947881:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576066.523713:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576066.971818:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1788576067.547724:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1788576067.547732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576067.547737:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576067.547744:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576067.547757:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12820 00000100:00080000:0.0:1788576067.547781:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576067.547789:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576067.547843:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350700 x1875444570042496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576067.547877:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576067.547881:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576067.547882:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576067.547886:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12820 00000100:00080000:0.0:1788576067.547890:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576067.547894:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576067.547905:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576067.547916:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350380 x1875444570042624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576067.547926:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576067.547929:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576067.547931:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576067.547935:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12820 00000100:00080000:0.0:1788576067.547938:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576067.547941:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576067.547946:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576067.547954:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353b80 x1875444570042752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576067.547962:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576067.547965:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576067.547967:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576067.547970:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12820 00000100:00080000:0.0:1788576067.547973:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576067.547976:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576067.547980:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576067.547989:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570042880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576067.547998:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576067.548001:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576067.548003:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576067.548006:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12820 00000100:00080000:0.0:1788576067.548008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576067.548011:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576067.548014:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576067.548022:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570043008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576067.548155:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570042496:192.168.202.157@tcp:250:kworker.0 00000800:00000100:3.0:1788576067.548223:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576067.548259:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff920a48350700 x1875444570042496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576067.548277:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350700 x1875444570042496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576067.548288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576067.548293:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576067.548297:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570042496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788576067.548304:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576067.548322:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576067.548332:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff920a48350380 x1875444570042624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576067.548343:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350380 x1875444570042624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576067.548350:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576067.548353:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576067.548356:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576067.548362:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576067.548376:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576067.548384:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff920a48353b80 x1875444570042752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576067.548394:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353b80 x1875444570042752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576067.548402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576067.548404:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576067.548407:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576067.548412:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576067.548427:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576067.548435:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff920a48353480 x1875444570042880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576067.548445:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570042880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576067.548453:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576067.548455:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576067.548458:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570042880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788576067.548463:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576067.548490:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576067.548498:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576067/real 1788576067] req@ffff920a48353800 x1875444570043008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576067.548510:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570043008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576067.548517:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576067.548519:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576067.548522:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576067.995822:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576068.571705:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576069.020842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576069.595712:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576070.043777:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576070.619699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576071.067836:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576071.646165:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576072.091871:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576072.667699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576072.667808:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576072.667815:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576072.667825:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576072.667835:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12826 00000100:00080000:1.0:1788576072.667845:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576072.667852:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576072.667888:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570043136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576072.667938:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570043136:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788576072.669860:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576072.669870:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576072.669873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576072.669883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12826 00000100:00080000:1.0:1788576072.669892:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576072.669899:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576072.669911:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576072.669940:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570043264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576072.674962:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043264:192.168.202.157@tcp:38:kworker.0 00000100:00080000:1.0:1788576072.675569:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576072.675576:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576072.675580:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576072.675600:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12826 00000100:00080000:1.0:1788576072.675606:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576072.675613:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576072.675623:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576072.675654:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570043392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576072.675676:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576072.675680:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576072.675682:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576072.675685:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12826 00000100:00080000:1.0:1788576072.675688:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576072.675691:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576072.675696:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576072.675708:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570043520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576072.675716:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576072.675720:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576072.675721:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576072.675726:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12826 00000100:00080000:1.0:1788576072.675728:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576072.675730:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576072.675734:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576072.675741:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570043648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576072.675972:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576072.676012:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576072.676033:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576073.116395:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576073.691821:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576074.140267:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576074.715848:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576075.163825:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576075.739877:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000400:00000100:3.0:1788576075.739977:0:2386:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:3.0:1788576075.740021:0:2386:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1788576075.740038:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788576075.740208:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576072/real 1788576075] req@ffff920a52890380 x1875444570043136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576097 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576075.740238:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570043136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576097 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576075.740252:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576075.740258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576075.740262:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570043136:192.168.202.157@tcp:250:kworker.0 00000800:00000100:3.0:1788576075.740324:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788576075.740345:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788576075.740353:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:3.0:1788576075.740362:0:2386:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788576075.740690:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576072/real 1788576075] req@ffff920a52891880 x1875444570043392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576075.740718:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570043392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576075.740724:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576075.740727:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576075.740730:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576075.740736:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576072/real 1788576075] req@ffff920a52893480 x1875444570043520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576075.740744:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570043520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576075.740749:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576075.740751:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576075.740753:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576075.740757:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576072/real 1788576075] req@ffff920a52891500 x1875444570043648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576075.740763:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570043648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576075.740767:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576075.740768:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576075.740770:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043648:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576075.740773:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576072/real 1788576075] req@ffff920a52891f80 x1875444570043264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576075.740791:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570043264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576075.740798:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576075.740801:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576075.740804:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043264:192.168.202.157@tcp:38:kworker.0 00010000:00010000:1.0:1788576076.187776:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576076.763727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576077.211754:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576077.787762:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576077.787805:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576077.787809:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576077.787812:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576077.787822:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12831 00000100:00080000:2.0:1788576077.787831:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576077.787838:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576077.787875:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444570043776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576077.787905:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576077.787909:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576077.787910:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576077.787915:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12831 00000100:00080000:2.0:1788576077.787918:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576077.787921:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576077.787934:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576077.787956:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444570043904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576077.787961:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570043776:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788576077.787972:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576077.787977:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576077.787978:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576077.787983:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12831 00000100:00080000:2.0:1788576077.787987:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576077.787991:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576077.787995:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576077.788010:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570044032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576077.788019:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576077.788022:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576077.788024:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576077.788027:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12831 00000100:00080000:2.0:1788576077.788031:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1788576077.788037:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788576077.788037:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576077.788046:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576077.788058:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570044160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576077.788067:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576077.788070:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576077.788072:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1788576077.788078:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576077/real 1788576077] req@ffff920a60ac7800 x1875444570043776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576077.788078:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12831 00000100:00080000:2.0:1788576077.788082:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576077.788089:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576077.788096:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576077.788102:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444570043776/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576077.788108:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570044288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576077.788123:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576077.788130:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576077.788135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570043776:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576077.788162:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043904:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576077.788187:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576077.788200:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576077/real 1788576077] req@ffff920a60ac4e00 x1875444570043904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576077.788213:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444570043904/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576077.788222:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576077.788226:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576077.788231:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570043904:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576077.788238:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044032:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576077.788258:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576077.788270:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576077/real 1788576077] req@ffff920a60ac5c00 x1875444570044032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576077.788280:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570044032/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576077.788287:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576077.788289:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576077.788291:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044032:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576077.788296:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044160:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576077.788314:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576077.788323:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576077/real 1788576077] req@ffff920a60ac4380 x1875444570044160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576077.788331:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570044160/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576077.788337:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576077.788340:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576077.788343:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044160:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576077.788347:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044288:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576077.788364:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576077.788374:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576077/real 1788576077] req@ffff920a60ac4700 x1875444570044288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576077.788383:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570044288/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576077.788394:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576077.788396:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576077.788399:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044288:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576078.236039:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576078.811852:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576079.262404:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576079.836763:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576080.284082:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576080.859741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576081.312537:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576081.883788:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576082.331994:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576082.907731:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576082.907836:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576082.907842:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576082.907846:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576082.907855:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12836 00000100:00080000:3.0:1788576082.907866:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576082.907873:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576082.907908:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570044416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576082.907938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576082.907942:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576082.907944:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576082.907949:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12836 00000100:00080000:3.0:1788576082.907952:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576082.907956:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576082.907965:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576082.907977:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570044544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576082.907995:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576082.907999:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576082.908001:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576082.908006:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12836 00000100:00080000:3.0:1788576082.908009:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576082.908012:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576082.908017:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576082.908026:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570044672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576082.908035:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576082.908038:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576082.908040:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576082.908043:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12836 00000100:00080000:3.0:1788576082.908046:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576082.908049:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576082.908053:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576082.908061:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570044800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576082.908071:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576082.908074:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576082.908076:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576082.908079:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12836 00000100:00080000:3.0:1788576082.908082:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576082.908085:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576082.908089:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576082.908097:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570044928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576082.908367:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570044416:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576082.908426:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576082.908456:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576082/real 1788576082] req@ffff920a529b0380 x1875444570044416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576082.908472:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570044416/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576082.908482:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576082.908486:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576082.908490:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570044416:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576082.908497:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044544:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576082.908515:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576082.908526:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576082/real 1788576082] req@ffff920a529b0700 x1875444570044544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576082.908537:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570044544/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576082.908545:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576082.908548:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576082.908551:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044544:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576082.908558:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044672:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576082.908573:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576082.908582:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576082/real 1788576082] req@ffff920a529b0000 x1875444570044672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576082.908629:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570044672/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576082.908638:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576082.908642:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576082.908645:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044672:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576082.908651:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044800:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576082.908669:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576082.908678:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576082/real 1788576082] req@ffff920a529b1f80 x1875444570044800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576082.908690:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570044800/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576082.908697:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576082.908700:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576082.908703:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044800:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576082.908709:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044928:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576082.908723:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576082.908732:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576082/real 1788576082] req@ffff920a529b3100 x1875444570044928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576082.908745:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570044928/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576082.908753:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576082.908756:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576082.908759:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570044928:192.168.202.157@tcp:8:kworker.0 00010000:00010000:2.0:1788576083.355903:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576083.935899:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576084.379857:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576084.955716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:2.0:1788576085.403838:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576085.980116:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576086.427829:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576087.005133:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576087.453233:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576088.027679:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576088.027772:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.027777:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576088.027780:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.027788:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12841 00000100:00080000:3.0:1788576088.027796:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.027801:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576088.027846:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570045056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576088.027909:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570045056:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576088.027960:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788576088.027986:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.027989:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576088.027990:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.027994:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12841 00000100:00080000:3.0:1788576088.027997:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.028001:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576088.028011:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576088.028025:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570045184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576088.028071:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff920a529b2a00 x1875444570045056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.028094:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570045056/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.028102:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.028104:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576088.028106:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570045056:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576088.028124:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045184:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576088.028149:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576088.028158:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff920a529b1c00 x1875444570045184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.028166:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570045184/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.028172:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.028174:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576088.028176:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045184:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788576088.028209:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.028212:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576088.028213:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.028217:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12841 00000100:00080000:3.0:1788576088.028220:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.028222:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576088.028226:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576088.028236:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570045312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576088.028242:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.028244:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576088.028245:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.028247:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12841 00000100:00080000:3.0:1788576088.028249:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.028251:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576088.028253:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576088.028259:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570045440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576088.028263:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576088.028265:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576088.028267:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576088.028269:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12841 00000100:00080000:3.0:1788576088.028271:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576088.028273:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576088.028275:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576088.028280:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570045568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576088.028336:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045312:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576088.028353:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576088.028361:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff920a529b3480 x1875444570045312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.028369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570045312/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.028374:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.028376:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576088.028378:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045312:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576088.028382:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045440:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576088.028400:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576088.028406:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff920a529b2d80 x1875444570045440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.028413:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570045440/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.028421:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.028423:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576088.028425:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045440:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576088.028428:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045568:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576088.028437:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576088.028442:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576088/real 1788576088] req@ffff920a529b1c00 x1875444570045568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576088.028448:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570045568/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576088.028452:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576088.028453:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576088.028455:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045568:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576088.475842:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576089.051914:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576089.499848:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576090.075721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576090.523803:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576091.100087:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576091.548886:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576092.123702:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576092.572705:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576093.147760:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576093.150103:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576093.150114:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576093.150119:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576093.150130:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12846 00000100:00080000:2.0:1788576093.150141:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576093.150149:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576093.150186:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570045696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576093.150218:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576093.150222:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576093.150224:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576093.150229:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12846 00000100:00080000:2.0:1788576093.150233:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576093.150237:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576093.150247:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576093.150261:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570045824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576093.150270:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576093.150274:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576093.150276:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576093.150281:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12846 00000100:00080000:2.0:1788576093.150284:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576093.150288:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576093.150294:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576093.150304:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570045952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576093.150314:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570045696:192.168.202.157@tcp:250:kworker.0 00000100:00080000:2.0:1788576093.150316:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576093.150324:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576093.150326:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576093.150330:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12846 00000100:00080000:2.0:1788576093.150334:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576093.150338:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576093.150343:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576093.150356:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570046080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576093.150365:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576093.150369:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576093.150371:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576093.150374:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12846 00000100:00080000:2.0:1788576093.150378:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576093.150382:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576093.150386:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:0.0:1788576093.150399:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576093.150399:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570046208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576093.150440:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff920a60ac4000 x1875444570045696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576093.150459:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570045696/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576093.150471:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576093.150476:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576093.150480:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570045696:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576093.150814:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045824:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576093.150857:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576093.150872:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff920a60ac6a00 x1875444570045824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576093.150886:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570045824/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576093.150895:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576093.150899:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576093.150902:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045824:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576093.150927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045952:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576093.150949:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576093.150962:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff920a60ac7100 x1875444570045952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576093.150975:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570045952/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576093.150983:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576093.151000:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576093.151004:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570045952:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576093.151011:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046080:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576093.151033:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576093.151044:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff920a60ac6300 x1875444570046080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576093.151055:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570046080/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576093.151063:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576093.151066:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576093.151069:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046080:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576093.151075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046208:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576093.151095:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576093.151106:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576093/real 1788576093] req@ffff920a60ac5180 x1875444570046208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576093.151117:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570046208/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576093.151125:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576093.151128:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576093.151131:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046208:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576093.596181:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576094.171757:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576094.620068:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576095.196815:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576095.643830:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576096.219802:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576096.667895:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576097.243753:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576097.691905:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576098.267834:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576098.267859:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576098.267865:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576098.267878:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12851 00000100:00080000:2.0:1788576098.267891:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576098.267900:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576098.267952:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444570046336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576098.267989:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576098.267992:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576098.267994:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576098.267999:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12851 00000100:00080000:2.0:1788576098.268002:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576098.268006:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576098.268018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576098.268029:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570046464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576098.268039:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576098.268042:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576098.268043:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576098.268105:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12851 00000100:00080000:2.0:1788576098.268110:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576098.268116:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576098.268121:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576098.268135:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570046592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576098.268144:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576098.268147:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576098.268149:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576098.268153:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12851 00000100:00080000:2.0:1788576098.268156:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576098.268160:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576098.268164:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576098.268173:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570046720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576098.268181:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576098.268184:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576098.268186:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576098.268189:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12851 00000100:00080000:2.0:1788576098.268192:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576098.268195:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576098.268199:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576098.268207:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570046848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576098.269210:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570046336:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576098.269289:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576098.269326:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff920a60ac5500 x1875444570046336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576098.269503:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788576098.269559:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444570046336/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576098.269570:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576098.269574:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576098.269577:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570046336:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576098.269760:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046464:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576098.269788:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576098.269801:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff920a60ac4a80 x1875444570046464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576098.269812:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570046464/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576098.269819:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576098.269822:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576098.269825:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046464:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576098.269831:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046592:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576098.269861:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576098.269871:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff920a60ac5180 x1875444570046592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576098.269886:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570046592/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576098.269893:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576098.269895:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576098.269897:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046592:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576098.269903:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046720:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576098.269922:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576098.269932:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff920a60ac6300 x1875444570046720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576098.269943:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570046720/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576098.269950:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576098.269953:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576098.269956:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046720:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576098.269961:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046848:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576098.269980:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576098.269991:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576098/real 1788576098] req@ffff920a60ac7100 x1875444570046848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576098.270002:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570046848/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576098.270015:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576098.270019:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576098.270022:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570046848:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576098.718024:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576099.291700:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576099.740204:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576100.317083:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576100.763900:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576101.340180:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576101.788009:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576102.363783:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576102.813173:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576103.387781:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576103.388459:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576103.388466:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576103.388470:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576103.388480:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12856 00000100:00080000:3.0:1788576103.388490:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576103.388498:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576103.388534:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570046976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576103.388565:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576103.388568:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576103.388571:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576103.388575:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12856 00000100:00080000:3.0:1788576103.388579:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576103.388614:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576103.388626:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576103.388640:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570047104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576103.388656:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576103.388659:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576103.388661:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576103.388665:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12856 00000100:00080000:3.0:1788576103.388669:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576103.388673:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576103.388678:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576103.388690:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570047232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576103.388700:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576103.388703:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576103.388705:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576103.388708:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12856 00000100:00080000:3.0:1788576103.388712:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576103.388716:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576103.388721:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576103.388732:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570047360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576103.388740:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576103.388743:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576103.388744:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576103.388747:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12856 00000100:00080000:3.0:1788576103.388751:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576103.388754:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576103.388759:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576103.388768:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570047488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576103.388816:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570046976:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576103.388885:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576103.388917:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff920a529b2a00 x1875444570046976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576103.388934:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570046976/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576103.388944:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576103.388948:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576103.388951:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570046976:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576103.388957:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047104:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576103.388976:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576103.388987:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff920a529b3100 x1875444570047104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576103.388998:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570047104/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576103.389006:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576103.389009:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576103.389012:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047104:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576103.389018:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047232:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576103.389033:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576103.389041:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff920a529b1f80 x1875444570047232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576103.389051:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570047232/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576103.389059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576103.389062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576103.389065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047232:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576103.389069:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047360:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576103.389083:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576103.389091:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff920a529b0000 x1875444570047360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576103.389101:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570047360/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576103.389109:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576103.389113:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576103.389116:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047360:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576103.389121:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047488:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576103.389777:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576103.389795:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576103/real 1788576103] req@ffff920a529b0700 x1875444570047488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576103.389808:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570047488/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576103.389816:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576103.389820:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576103.389823:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047488:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576103.835920:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576104.411732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576104.859785:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576105.435977:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576105.883771:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576106.459737:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576106.909464:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576107.483791:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576107.931816:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576108.507725:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576108.507743:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576108.507751:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576108.507755:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576108.507764:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12861 00000100:00080000:2.0:1788576108.507776:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576108.507784:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576108.507824:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570047616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576108.507859:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576108.507863:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576108.507873:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576108.507878:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12861 00000100:00080000:2.0:1788576108.507882:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576108.507886:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576108.507895:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576108.507908:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570047744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576108.507916:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576108.507920:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576108.507921:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576108.507925:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12861 00000100:00080000:2.0:1788576108.507928:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576108.507931:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576108.507936:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576108.507945:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570047872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576108.507953:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576108.507957:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576108.507958:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576108.507962:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12861 00000100:00080000:2.0:1788576108.507966:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576108.507969:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576108.507973:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576108.507981:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570048000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576108.507989:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576108.507992:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576108.507994:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576108.507998:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12861 00000100:00080000:2.0:1788576108.508001:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576108.508004:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576108.508009:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576108.508017:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570048128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576108.508092:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570047616:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576108.508156:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576108.508190:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576108/real 1788576108] req@ffff920a60ac6a00 x1875444570047616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576108.508207:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570047616/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576108.508217:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576108.508221:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576108.508225:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570047616:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576108.508232:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047744:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576108.508256:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576108.508268:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576108/real 1788576108] req@ffff920a60ac4000 x1875444570047744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576108.508280:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570047744/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576108.508288:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576108.508291:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576108.508294:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047744:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576108.508300:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047872:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576108.508317:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576108.508326:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576108/real 1788576108] req@ffff920a60ac4700 x1875444570047872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576108.508337:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570047872/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576108.508344:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576108.508347:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576108.508350:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570047872:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576108.508356:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048000:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576108.508371:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576108.508379:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576108/real 1788576108] req@ffff920a60ac4380 x1875444570048000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576108.508395:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570048000/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576108.508402:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576108.508405:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576108.508408:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048000:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576108.508413:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048128:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576108.508429:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576108.508439:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576108/real 1788576108] req@ffff920a60ac5c00 x1875444570048128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576108.508449:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570048128/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576108.508456:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576108.508458:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576108.508461:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048128:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576108.955759:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576109.531732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576109.982151:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576110.555844:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576111.003841:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576111.579717:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576112.027839:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576112.603715:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576113.052973:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576113.627749:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576113.627762:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576113.627767:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576113.627786:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12867 00000100:00080000:2.0:1788576113.627805:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576113.627813:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576113.627859:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4e00 x1875444570048256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576113.627911:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576113.627915:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576113.627916:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576113.627920:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12867 00000100:00080000:2.0:1788576113.627924:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576113.627927:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576113.627939:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576113.627949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7800 x1875444570048384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576113.627958:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576113.627961:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576113.627963:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576113.627967:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12867 00000100:00080000:2.0:1788576113.627969:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576113.627972:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576113.627975:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576113.627985:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5c00 x1875444570048512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576113.627993:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576113.627996:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576113.627998:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576113.628001:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12867 00000100:00080000:2.0:1788576113.628003:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576113.628006:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576113.628011:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576113.628018:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4380 x1875444570048640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576113.628026:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576113.628028:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576113.628030:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576113.628032:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12867 00000100:00080000:2.0:1788576113.628035:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576113.628038:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576113.628041:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576113.628048:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4700 x1875444570048768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576113.628233:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570048256:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576113.628308:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576113.628380:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788576113.631985:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576113/real 1788576113] req@ffff920a60ac4e00 x1875444570048256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576113.632026:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4e00 x1875444570048256/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576113.632037:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576113.632042:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576113.632046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570048256:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576113.632058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048384:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576113.632121:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576113.632144:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576113/real 1788576113] req@ffff920a60ac7800 x1875444570048384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576113.632157:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7800 x1875444570048384/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576113.632166:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576113.632169:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576113.632172:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048384:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576113.632180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048512:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576113.632202:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576113.632215:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576113/real 1788576113] req@ffff920a60ac5c00 x1875444570048512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576113.632227:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5c00 x1875444570048512/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576113.632241:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576113.632245:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576113.632248:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048512:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576113.632255:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048640:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576113.632277:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576113.632288:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576113/real 1788576113] req@ffff920a60ac4380 x1875444570048640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576113.632300:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4380 x1875444570048640/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576113.632307:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576113.632310:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576113.632314:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048640:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576113.632320:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048768:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576113.632340:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576113.632350:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576113/real 1788576113] req@ffff920a60ac4700 x1875444570048768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576113.632362:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4700 x1875444570048768/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576113.632370:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576113.632373:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576113.632377:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570048768:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576114.075851:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576114.654107:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576115.100627:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576115.675710:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576116.123792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576116.699721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576117.147914:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576117.723732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576118.171777:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576118.747717:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788576118.747721:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:2.0:1788576118.747731:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576118.747737:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576118.747749:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12872 00000100:00080000:2.0:1788576118.747760:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576118.747769:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576118.747815:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4000 x1875444570048896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576118.747918:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570048896:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576118.747994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:2.0:1788576118.748058:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576118.748061:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576118.748062:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576118.748065:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12872 00000100:00080000:2.0:1788576118.748069:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576118.748072:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576118.748081:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576118.748092:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6a00 x1875444570049024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576118.750685:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576118/real 1788576118] req@ffff920a60ac4000 x1875444570048896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576118.750718:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4000 x1875444570048896/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576118.750729:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576118.750734:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576118.750738:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570048896:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576118.750774:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049024:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576118.750836:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576118.750854:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576118/real 1788576118] req@ffff920a60ac6a00 x1875444570049024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576118.750866:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6a00 x1875444570049024/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576118.750874:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576118.750878:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576118.750881:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049024:192.168.202.157@tcp:38:kworker.0 00000100:00080000:2.0:1788576118.751986:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576118.751993:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576118.751997:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576118.752006:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12872 00000100:00080000:2.0:1788576118.752011:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576118.752016:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576118.752027:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576118.752060:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570049152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576118.752074:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576118.752077:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576118.752079:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576118.752082:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12872 00000100:00080000:2.0:1788576118.752086:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576118.752090:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576118.752095:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576118.752108:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570049280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576118.752123:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576118.752126:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576118.752128:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576118.752131:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12872 00000100:00080000:2.0:1788576118.752134:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576118.752138:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576118.752143:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576118.752152:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570049408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576118.752322:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049152:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576118.752373:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576118.752391:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576118/real 1788576118] req@ffff920a60ac7100 x1875444570049152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576118.752405:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570049152/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576118.752414:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576118.752417:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576118.752421:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049152:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576118.752426:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049280:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576118.752446:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576118.752456:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576118/real 1788576118] req@ffff920a60ac6300 x1875444570049280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576118.752467:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570049280/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576118.752475:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576118.752478:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576118.752480:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049280:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576118.752486:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049408:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576118.752502:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576118.752522:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576118/real 1788576118] req@ffff920a60ac5180 x1875444570049408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576118.752533:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570049408/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576118.752542:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576118.752545:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576118.752548:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049408:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576119.196613:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576119.771711:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576120.219936:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576120.795694:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576121.244001:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576121.820297:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576122.267806:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576122.843973:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576123.291790:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576123.867703:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:0.0:1788576123.867857:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576123.867863:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1788576123.867867:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576123.867877:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12877 00000100:00080000:0.0:1788576123.867887:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576123.867894:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:0.0:1788576123.867945:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351c00 x1875444570049536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576123.867976:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576123.867980:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1788576123.867982:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576123.867986:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12877 00000100:00080000:0.0:1788576123.867990:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576123.867994:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576123.868004:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576123.868015:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48351500 x1875444570049664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576123.868024:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576123.868027:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:0.0:1788576123.868029:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576123.868033:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12877 00000100:00080000:0.0:1788576123.868035:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576123.868039:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576123.868043:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1788576123.868053:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48350a80 x1875444570049792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576123.868062:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576123.868065:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1788576123.868067:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576123.868070:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12877 00000100:00080000:0.0:1788576123.868073:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1788576123.868075:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570049536:192.168.202.157@tcp:250:kworker.0 00000100:00080000:0.0:1788576123.868079:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576123.868086:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576123.868100:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353800 x1875444570049920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576123.868109:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1788576123.868112:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1788576123.868113:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1788576123.868117:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12877 00000100:00080000:0.0:1788576123.868133:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1788576123.868138:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:0.0:1788576123.868143:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000800:00000100:1.0:1788576123.868143:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576123.868158:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a48353480 x1875444570050048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576123.868187:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576123/real 1788576123] req@ffff920a48351c00 x1875444570049536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576123.868202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351c00 x1875444570049536/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576123.868209:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576123.868212:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576123.868215:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570049536:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788576123.868220:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049664:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788576123.868236:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576123.868244:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576123/real 1788576123] req@ffff920a48351500 x1875444570049664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576123.868251:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48351500 x1875444570049664/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576123.868256:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576123.868258:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576123.868260:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48351500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049664:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788576123.868278:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049792:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788576123.868289:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576123.868297:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576123/real 1788576123] req@ffff920a48350a80 x1875444570049792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576123.868309:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48350a80 x1875444570049792/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576123.868314:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576123.868316:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576123.868318:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48350a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049792:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788576123.868321:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049920:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788576123.868330:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576123.868335:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576123/real 1788576123] req@ffff920a48353800 x1875444570049920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576123.868341:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353800 x1875444570049920/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576123.868346:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576123.868347:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576123.868349:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570049920:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788576123.868352:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050048:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788576123.868360:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576123.868368:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576123/real 1788576123] req@ffff920a48353480 x1875444570050048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576123.868379:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a48353480 x1875444570050048/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576123.868391:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576123.868394:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576123.868397:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a48353480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050048:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576124.315958:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576124.891749:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576125.339843:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576125.915719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576126.364095:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576126.940668:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576127.387921:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576127.965061:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576128.413773:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576128.987801:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576128.988621:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576128.988629:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576128.988633:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576128.988643:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12882 00000100:00080000:3.0:1788576128.988653:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576128.988665:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576128.988701:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570050176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576128.988725:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576128.988729:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576128.988730:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576128.988734:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12882 00000100:00080000:3.0:1788576128.988737:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576128.988741:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576128.988750:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576128.988761:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570050304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576128.988770:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576128.988773:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576128.988775:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576128.988779:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12882 00000100:00080000:3.0:1788576128.988782:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576128.988785:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576128.988789:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576128.988798:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570050432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576128.988806:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576128.988809:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576128.988810:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576128.988814:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12882 00000100:00080000:3.0:1788576128.988817:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576128.988820:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576128.988824:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576128.988833:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570050560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576128.988839:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576128.988842:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576128.988844:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576128.988847:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12882 00000100:00080000:3.0:1788576128.988850:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576128.988854:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576128.988859:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576128.988868:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570050688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576128.988874:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570050176:192.168.202.157@tcp:250:kworker.0 00000800:00000100:1.0:1788576128.988954:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576128.988985:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576128/real 1788576128] req@ffff920a529b0380 x1875444570050176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576128.989000:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570050176/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576128.989009:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576128.989013:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576128.989017:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570050176:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788576128.989023:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050304:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788576128.989043:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576128.989056:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576128/real 1788576128] req@ffff920a529b3b80 x1875444570050304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576128.989072:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570050304/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576128.989080:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576128.989083:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576128.989085:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050304:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788576128.989091:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050432:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788576128.989109:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576128.989189:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576128/real 1788576128] req@ffff920a529b0e00 x1875444570050432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576128.989202:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570050432/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576128.989210:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576128.989213:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576128.989217:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050432:192.168.202.157@tcp:38:kworker.0 00000100:00100000:1.0:1788576128.989223:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050560:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788576128.989246:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576128.989257:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576128/real 1788576128] req@ffff920a529b2300 x1875444570050560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576128.989268:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570050560/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576128.989280:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576128.989283:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576128.989285:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050560:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788576128.989309:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050688:192.168.202.157@tcp:8:kworker.0 00000800:00000100:1.0:1788576128.989327:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576128.989338:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576128/real 1788576128] req@ffff920a529b1500 x1875444570050688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576128.989348:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570050688/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576153 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576128.989355:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576128.989358:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576128.989361:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050688:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576129.435918:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576130.012360:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576130.462105:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576131.035999:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576131.484116:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576132.060198:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576132.508076:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576133.083798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576133.531869:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576134.107673:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576134.107686:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1788576134.107687:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576134.107690:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576134.107716:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12887 00000100:00080000:3.0:1788576134.107728:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576134.107735:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576134.107771:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570050816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576134.107793:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576134.107795:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576134.107797:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576134.107800:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12887 00000100:00080000:3.0:1788576134.107802:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576134.107805:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576134.107813:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576134.107827:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570050944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576134.107838:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576134.107841:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576134.107843:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576134.107848:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12887 00000100:00080000:3.0:1788576134.107855:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1788576134.107856:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570050816:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576134.107862:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576134.107869:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576134.107885:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570051072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576134.107895:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576134.107899:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576134.107901:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576134.107906:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12887 00000100:00080000:3.0:1788576134.107910:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576134.107915:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576134.107920:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576134.107947:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570051200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576134.107956:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576134.107959:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576134.107962:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576134.107967:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12887 00000100:00080000:3.0:1788576134.107971:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576134.107976:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576134.107981:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576134.107992:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570051328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1788576134.107995:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576134.108033:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff920a529b1180 x1875444570050816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576134.108049:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570050816/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576134.108059:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576134.108062:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576134.108065:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570050816:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576134.108072:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050944:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576134.108096:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576134.108108:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff920a529b1500 x1875444570050944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576134.108120:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570050944/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576134.108128:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576134.108132:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576134.108135:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570050944:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576134.108167:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051072:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576134.108183:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576134.108191:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff920a529b2300 x1875444570051072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576134.108198:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570051072/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576134.108203:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576134.108205:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576134.108207:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051072:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576134.108211:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051200:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576134.108221:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576134.108226:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff920a529b0e00 x1875444570051200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576134.108232:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570051200/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576134.108237:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576134.108238:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576134.108240:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051200:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576134.108243:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051328:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576134.108253:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576134.108258:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576134/real 1788576134] req@ffff920a529b3b80 x1875444570051328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576134.108265:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570051328/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576134.108269:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576134.108271:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576134.108273:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051328:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576134.556381:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576135.134130:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576135.579837:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576136.158706:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576136.604299:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576137.179897:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576137.628174:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576138.203772:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576138.652695:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576139.227727:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1788576139.227735:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576139.227741:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576139.227748:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576139.227761:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12892 00000100:00080000:3.0:1788576139.227773:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576139.227782:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576139.227827:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570051456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576139.227863:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576139.227876:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576139.227878:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576139.227883:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12892 00000100:00080000:3.0:1788576139.227887:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576139.227892:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576139.227903:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576139.227920:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570051584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576139.227926:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570051456:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576139.227938:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576139.227943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576139.227955:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576139.227961:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12892 00000100:00080000:3.0:1788576139.227964:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576139.227969:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576139.227975:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576139.227991:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570051712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576139.228003:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576139.228007:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576139.228009:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576139.228022:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12892 00000100:00080000:3.0:1788576139.228027:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576139.228033:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576139.228041:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576139.228053:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570051840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576139.228058:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051584:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788576139.228087:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576139.228092:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576139.228094:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576139.228100:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12892 00000100:00080000:3.0:1788576139.228104:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576139.228108:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576139.228113:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576139.228123:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570051968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576139.228134:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576139.228177:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576139.228204:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051968:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576139.675886:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576140.251750:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576140.699920:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576141.278150:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576141.727266:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1788576142.300153:0:2387:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.202.157:988 00000400:00000100:2.0:1788576142.300204:0:2387:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.202.157@tcp at host 192.168.202.157 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1788576142.300222:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576142.300258:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576142.300268:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576142.300277:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000800:00000100:2.0:1788576142.300286:0:2387:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.202.157@tcp 00000100:00100000:0.0:1788576142.307432:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576142] req@ffff920a529b1f80 x1875444570051840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576142.307456:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788576142.307465:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570051840/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576142.307477:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576142.307482:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576142.307487:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051840:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576142.307496:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576142] req@ffff920a529b3100 x1875444570051968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576142.307507:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570051968/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576142.307514:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576142.307518:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576142.307521:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051968:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576142.307531:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576142] req@ffff920a529b0700 x1875444570051584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576142.307544:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570051584/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576142.307552:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576142.307556:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576142.307559:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051584:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576142.307567:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576142] req@ffff920a529b0000 x1875444570051712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576142.307578:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570051712/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576142.307605:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576142.307611:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576142.307616:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570051712:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576142.307625:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576139/real 1788576142] req@ffff920a529b0380 x1875444570051456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576142.307639:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570051456/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576142.307647:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576142.307652:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576142.307656:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570051456:192.168.202.157@tcp:250:kworker.0 00010000:00010000:1.0:1788576142.747796:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576143.324096:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576143.771784:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576144.347869:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576144.347880:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576144.347885:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576144.347896:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12897 00000100:00100000:1.0:1788576144.347904:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576144.347907:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.347915:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576144.347949:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570052096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576144.347993:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576144.347997:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576144.347999:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576144.348004:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12897 00000100:00080000:3.0:1788576144.348008:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.348013:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576144.348023:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576144.348036:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570052224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576144.348046:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576144.348050:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576144.348051:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576144.348055:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12897 00000100:00080000:3.0:1788576144.348059:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.348062:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576144.348067:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576144.348077:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570052352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576144.348090:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576144.348093:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576144.348094:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576144.348098:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12897 00000100:00080000:3.0:1788576144.348101:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.348104:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576144.348109:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576144.348118:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570052480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576144.348126:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576144.348128:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576144.348130:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576144.348133:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12897 00000100:00080000:3.0:1788576144.348136:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576144.348139:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576144.348144:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576144.348155:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570052608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576144.353680:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570052096:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576144.353774:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576144.353813:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff920a529b2a00 x1875444570052096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576144.353831:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570052096/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.353842:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.353846:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576144.353850:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570052096:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576144.353858:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052224:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576144.353883:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576144.353894:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff920a529b1c00 x1875444570052224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576144.353905:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570052224/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.353913:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.353916:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576144.353919:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052224:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576144.353927:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052352:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576144.353948:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576144.353971:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff920a529b2d80 x1875444570052352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576144.353987:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570052352/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.353995:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.353998:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576144.354001:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052352:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576144.354007:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052480:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576144.354027:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576144.354037:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff920a529b3480 x1875444570052480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576144.354050:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570052480/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.354056:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.354059:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576144.354062:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052480:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576144.354068:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052608:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576144.354086:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576144.354096:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576144/real 1788576144] req@ffff920a529b0380 x1875444570052608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576144.354106:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570052608/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576144.354118:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576144.354121:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576144.354124:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052608:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576144.795867:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576145.371791:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576145.821885:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576146.395878:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576146.844633:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576147.420435:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576147.867796:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576148.444092:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576148.891900:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576149.467747:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576149.467761:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576149.467766:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576149.467779:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12902 00000100:00080000:3.0:1788576149.467792:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576149.467801:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576149.467850:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570052736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576149.467890:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576149.467894:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576149.467896:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576149.467900:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12902 00000100:00080000:3.0:1788576149.467903:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576149.467907:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576149.467919:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576149.467934:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570052864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576149.467944:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576149.467947:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576149.467948:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576149.467952:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12902 00000100:00080000:3.0:1788576149.467956:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576149.467960:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576149.467964:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576149.467987:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570052992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576149.467997:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576149.468001:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576149.468003:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576149.468006:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12902 00000100:00080000:3.0:1788576149.468010:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576149.468014:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576149.468019:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576149.468029:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570053120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576149.468037:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576149.468040:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576149.468042:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576149.468045:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12902 00000100:00080000:3.0:1788576149.468048:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576149.468052:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576149.468057:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576149.468064:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570053248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576149.468180:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570052736:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576149.468260:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576149.468302:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576149/real 1788576149] req@ffff920a529b0000 x1875444570052736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576149.468322:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570052736/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576149.468334:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576149.468338:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576149.468347:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570052736:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788576149.468354:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:2.0:1788576149.468356:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052864:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576149.468387:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576149.468398:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576149/real 1788576149] req@ffff920a529b0700 x1875444570052864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576149.468410:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570052864/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576149.468418:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576149.468422:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576149.468425:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052864:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576149.468432:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052992:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576149.468464:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576149.468475:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576149/real 1788576149] req@ffff920a529b3100 x1875444570052992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576149.468487:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570052992/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576149.468496:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576149.468499:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576149.468502:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570052992:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576149.468508:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053120:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576149.468526:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576149.468535:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576149/real 1788576149] req@ffff920a529b1f80 x1875444570053120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576149.468546:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570053120/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576149.468555:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576149.468558:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576149.468561:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053120:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576149.468567:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053248:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576149.468583:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576149.468622:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576149/real 1788576149] req@ffff920a529b3b80 x1875444570053248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576149.468633:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570053248/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576149.468641:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576149.468644:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576149.468647:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053248:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788576149.915812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576150.494781:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576150.939822:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576151.515963:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576151.964205:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576152.539756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576152.988066:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1788576153.563867:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576154.012058:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576154.587741:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576154.587755:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576154.587761:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576154.587776:0:94873:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12907 00000100:00080000:3.0:1788576154.587789:0:94873:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576154.587797:0:94873:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576154.587847:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570053376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576154.587872:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576154.587876:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576154.587878:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576154.587888:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12907 00000100:00080000:3.0:1788576154.587891:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576154.587896:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576154.587908:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576154.587922:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570053504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576154.587929:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570053376:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576154.587935:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576154.587943:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576154.587945:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576154.587950:0:94873:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12907 00000100:00080000:3.0:1788576154.587953:0:94873:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576154.587956:0:94873:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576154.587961:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576154.587973:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570053632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576154.587982:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1788576154.588023:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576154.588061:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576154/real 1788576154] req@ffff920a529b0e00 x1875444570053376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576154.588080:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570053376/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576154.588092:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576154.588096:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576154.588100:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570053376:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576154.588128:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053504:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576154.588176:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576154.588202:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576154/real 1788576154] req@ffff920a529b2300 x1875444570053504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576154.588228:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570053504/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576154.588249:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576154.588256:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576154.588263:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053504:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576154.588340:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053632:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576154.588381:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576154.588410:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576154/real 1788576154] req@ffff920a529b1500 x1875444570053632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576154.588436:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570053632/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576154.588449:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576154.588456:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576154.588465:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053632:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788576154.588539:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576154.588542:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576154.588548:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12907 00000100:00080000:3.0:1788576154.588554:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576154.588559:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576154.588566:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576154.588583:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1180 x1875444570053760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576154.588635:0:94873:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576154.588638:0:94873:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576154.588640:0:94873:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576154.588644:0:94873:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12907 00000100:00080000:3.0:1788576154.588648:0:94873:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576154.588651:0:94873:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576154.588656:0:94873:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576154.588668:0:94873:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1500 x1875444570053888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576154.588779:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053760:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576154.588828:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576154.588862:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576154/real 1788576154] req@ffff920a529b1180 x1875444570053760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576154.588889:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1180 x1875444570053760/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576154.588903:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576154.588909:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576154.588917:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053760:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576154.588934:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053888:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576154.588974:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576154.589002:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576154/real 1788576154] req@ffff920a529b1500 x1875444570053888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576154.589012:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1500 x1875444570053888/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576154.589019:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576154.589022:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576154.589025:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570053888:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788576154.589835:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576155.036148:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576155.611727:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576156.059812:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576156.635870:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576157.083819:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576157.660849:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576158.109325:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576158.683726:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576159.131909:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576159.707756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576159.707763:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576159.707774:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576159.707778:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576159.707787:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12913 00000100:00080000:3.0:1788576159.707799:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576159.707805:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576159.707841:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2300 x1875444570054016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576159.707872:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576159.707877:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576159.707879:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576159.707883:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12913 00000100:00080000:3.0:1788576159.707887:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576159.707890:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576159.707900:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576159.707917:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0e00 x1875444570054144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576159.707927:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576159.707931:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576159.707933:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576159.707937:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12913 00000100:00080000:3.0:1788576159.707941:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1788576159.707947:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570054016:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576159.707949:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576159.707960:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576159.707972:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570054272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576159.707984:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576159.707988:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576159.707989:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576159.707993:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12913 00000100:00080000:3.0:1788576159.708008:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576159.708012:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576159.708018:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576159.708029:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570054400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576159.708039:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576159.708042:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576159.708044:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576159.708047:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12913 00000100:00080000:3.0:1788576159.708051:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1788576159.708058:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788576159.708061:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576159.708069:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576159.708080:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570054528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576159.708116:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576159/real 1788576159] req@ffff920a529b2300 x1875444570054016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576159.708138:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2300 x1875444570054016/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576159.708151:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576159.708155:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576159.708160:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570054016:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576159.708168:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054144:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576159.708194:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576159.708206:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576159/real 1788576159] req@ffff920a529b0e00 x1875444570054144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576159.708220:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0e00 x1875444570054144/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576159.708230:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576159.708234:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576159.708239:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054144:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576159.708278:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054272:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576159.708297:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576159.708308:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576159/real 1788576159] req@ffff920a529b3b80 x1875444570054272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576159.708320:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570054272/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576159.708328:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576159.708332:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576159.708335:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054272:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576159.708342:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054400:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576159.708359:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576159.708369:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576159/real 1788576159] req@ffff920a529b1f80 x1875444570054400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576159.708378:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570054400/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576159.708385:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576159.708388:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576159.708390:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054400:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576159.708395:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054528:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576159.708411:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576159.708421:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576159/real 1788576159] req@ffff920a529b3100 x1875444570054528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576159.708431:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570054528/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576159.708438:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576159.708441:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576159.708444:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054528:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788576160.155778:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576160.734100:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576161.179757:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576161.755855:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576162.204867:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576162.779908:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576163.228082:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576163.803724:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576164.252554:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576164.827985:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576164.829161:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576164.829180:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576164.829184:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576164.829195:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12918 00000100:00080000:1.0:1788576164.829206:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576164.829215:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576164.829252:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570054656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576164.829284:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576164.829288:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576164.829290:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576164.829294:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12918 00000100:00080000:1.0:1788576164.829297:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576164.829301:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576164.829311:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576164.829322:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570054784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576164.829331:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576164.829335:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576164.829337:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576164.829341:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12918 00000100:00080000:1.0:1788576164.829344:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576164.829347:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576164.829351:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576164.829360:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570054912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576164.829368:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576164.829371:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576164.829372:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576164.829376:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12918 00000100:00080000:1.0:1788576164.829378:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576164.829382:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576164.829386:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576164.829394:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570055040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576164.829402:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576164.829405:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576164.829406:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576164.829410:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12918 00000100:00080000:1.0:1788576164.829413:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576164.829416:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576164.829419:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576164.829429:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570055168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576164.829581:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570054656:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576164.829665:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576164.829699:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576164/real 1788576164] req@ffff920a52890e00 x1875444570054656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576164.829715:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570054656/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576164.829729:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576164.829733:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576164.829737:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570054656:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576164.829743:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054784:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576164.829765:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576164.829777:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576164/real 1788576164] req@ffff920a52893800 x1875444570054784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576164.829788:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570054784/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576164.829795:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576164.829798:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576164.829800:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054784:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576164.829806:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054912:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576164.829825:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576164.829835:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576164/real 1788576164] req@ffff920a52892d80 x1875444570054912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576164.829845:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570054912/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576164.829852:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576164.829855:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576164.829858:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570054912:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576164.829864:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055040:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576164.829884:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576164.829893:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576164/real 1788576164] req@ffff920a52892300 x1875444570055040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576164.829904:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570055040/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576164.829912:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576164.829915:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576164.829918:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055040:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576164.829923:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055168:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576164.829940:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576164.829950:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576164/real 1788576164] req@ffff920a52891180 x1875444570055168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576164.829960:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570055168/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576164.829966:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576164.829969:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576164.829972:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055168:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576165.275774:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576165.852299:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576166.302081:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576166.875707:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576167.324784:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576167.899756:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576168.349383:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576168.923741:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576169.371842:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576169.947918:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576169.948375:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576169.948383:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576169.948386:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576169.948394:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12923 00000100:00080000:1.0:1788576169.948403:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576169.948409:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576169.948436:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893100 x1875444570055296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576169.948460:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576169.948462:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576169.948464:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576169.948467:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12923 00000100:00080000:1.0:1788576169.948469:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576169.948471:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576169.948478:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576169.948486:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890700 x1875444570055424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576169.948493:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576169.948496:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576169.948499:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576169.948508:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12923 00000100:00080000:1.0:1788576169.948517:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576169.948521:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576169.948524:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576169.948530:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891180 x1875444570055552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576169.948535:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576169.948537:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576169.948538:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576169.948541:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12923 00000100:00080000:1.0:1788576169.948542:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576169.948544:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576169.948547:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576169.948552:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892300 x1875444570055680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576169.948557:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576169.948558:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576169.948560:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576169.948562:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12923 00000100:00080000:1.0:1788576169.948565:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576169.948567:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576169.948569:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576169.948576:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892d80 x1875444570055808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576169.948647:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570055296:192.168.202.157@tcp:250:kworker.0 00000800:00000100:3.0:1788576169.949448:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576169.949496:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576169/real 1788576169] req@ffff920a52893100 x1875444570055296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576169.949521:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893100 x1875444570055296/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576169.949535:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576169.949540:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576169.949545:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570055296:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788576169.949555:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055424:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576169.949618:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576169.949633:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576169/real 1788576169] req@ffff920a52890700 x1875444570055424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576169.949647:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890700 x1875444570055424/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576169.949656:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576169.949660:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576169.949663:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055424:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576169.949671:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055552:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576169.949693:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576169.949704:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576169/real 1788576169] req@ffff920a52891180 x1875444570055552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576169.949716:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891180 x1875444570055552/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576169.949724:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576169.949728:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576169.949799:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055552:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576169.949807:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055680:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576169.949833:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576169.949846:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576169/real 1788576169] req@ffff920a52892300 x1875444570055680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576169.949865:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892300 x1875444570055680/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576169.949874:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576169.949878:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576169.949882:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055680:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788576169.949891:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055808:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576169.949914:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576169.949932:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576169/real 1788576169] req@ffff920a52892d80 x1875444570055808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576169.949946:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892d80 x1875444570055808/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576194 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576169.949956:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576169.949960:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576169.949964:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570055808:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788576170.396083:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576170.971798:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576171.420895:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576171.995799:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576172.444027:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576173.019872:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576173.467852:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576174.043733:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576174.493319:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576175.067983:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576175.068063:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576175.068068:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576175.068072:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576175.068082:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12928 00000100:00080000:3.0:1788576175.068092:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576175.068099:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576175.068136:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570055936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576175.068207:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570055936:192.168.202.157@tcp:250:kworker.0 00000800:00000100:2.0:1788576175.068268:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788576175.068302:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576175.068306:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576175.068308:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576175.068313:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12928 00000100:00080000:3.0:1788576175.068318:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576175.068322:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576175.068344:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576175.068358:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570056064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576175.078770:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576175/real 1788576175] req@ffff920a529b0700 x1875444570055936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576175.078799:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570055936/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576175.078811:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576175.078816:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576175.078821:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570055936:192.168.202.157@tcp:250:kworker.0 00000100:00100000:2.0:1788576175.078852:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056064:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576175.078919:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576175.078937:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576175/real 1788576175] req@ffff920a529b0000 x1875444570056064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576175.078950:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570056064/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576175.078970:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576175.078975:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576175.078978:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056064:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788576175.079465:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576175.079471:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576175.079474:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576175.079479:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12928 00000100:00080000:3.0:1788576175.079485:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576175.079492:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576175.079503:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576175.079528:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570056192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576175.079542:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576175.079545:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576175.079547:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576175.079550:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12928 00000100:00080000:3.0:1788576175.079555:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576175.079559:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576175.079564:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576175.079575:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570056320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576175.079610:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576175.079614:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576175.079615:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576175.079619:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12928 00000100:00080000:3.0:1788576175.079623:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576175.079626:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576175.079631:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576175.079642:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570056448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576175.079820:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056192:192.168.202.157@tcp:38:kworker.0 00000800:00000100:2.0:1788576175.079847:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576175.079860:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576175/real 1788576175] req@ffff920a529b0380 x1875444570056192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576175.079875:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570056192/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576175.079883:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576175.079887:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576175.079890:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056192:192.168.202.157@tcp:38:kworker.0 00000100:00100000:2.0:1788576175.079896:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056320:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576175.079917:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576175.079928:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576175/real 1788576175] req@ffff920a529b3480 x1875444570056320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576175.079940:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570056320/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576175.079948:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576175.079951:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576175.079954:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056320:192.168.202.157@tcp:8:kworker.0 00000100:00100000:2.0:1788576175.079973:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056448:192.168.202.157@tcp:8:kworker.0 00000800:00000100:2.0:1788576175.079994:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:2.0:1788576175.080018:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576175/real 1788576175] req@ffff920a529b2d80 x1875444570056448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1788576175.080031:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570056448/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1788576175.080044:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1788576175.080048:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:2.0:1788576175.080051:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056448:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576175.516578:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576176.091814:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576176.540246:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576177.116199:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576177.564573:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576178.139830:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576178.590247:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576179.163719:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576179.611800:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1788576180.187963:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576180.187974:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1788576180.187978:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576180.187987:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12933 00000100:00080000:2.0:1788576180.188007:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576180.188014:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:2.0:1788576180.188047:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac4a80 x1875444570056576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576180.188077:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576180.188088:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1788576180.188096:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576180.188100:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12933 00000100:00080000:2.0:1788576180.188104:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576180.188108:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576180.188117:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576180.188130:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5500 x1875444570056704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576180.188146:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576180.188149:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:2.0:1788576180.188151:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576180.188155:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12933 00000100:00080000:2.0:1788576180.188158:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576180.188162:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576180.188166:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1788576180.188178:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac5180 x1875444570056832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576180.188187:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576180.188190:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1788576180.188192:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576180.188195:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12933 00000100:00080000:2.0:1788576180.188199:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576180.188203:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576180.188207:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576180.188216:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac6300 x1875444570056960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1788576180.188224:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1788576180.188226:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1788576180.188228:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1788576180.188231:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12933 00000100:00080000:2.0:1788576180.188234:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1788576180.188238:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:2.0:1788576180.188242:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:2.0:1788576180.188251:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a60ac7100 x1875444570057088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1788576180.188397:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570056576:192.168.202.157@tcp:250:kworker.0 00000800:00000100:3.0:1788576180.188474:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576180.188515:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576180/real 1788576180] req@ffff920a60ac4a80 x1875444570056576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576180.188532:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac4a80 x1875444570056576/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576180.188541:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576180.188545:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576180.188548:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac4a80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570056576:192.168.202.157@tcp:250:kworker.0 00000100:00100000:3.0:1788576180.188555:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056704:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576180.188578:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576180.188607:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576180/real 1788576180] req@ffff920a60ac5500 x1875444570056704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576180.188619:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5500 x1875444570056704/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576180.188626:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576180.188629:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576180.188632:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056704:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576180.188638:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056832:192.168.202.157@tcp:38:kworker.0 00000800:00000100:3.0:1788576180.188657:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576180.188666:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576180/real 1788576180] req@ffff920a60ac5180 x1875444570056832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576180.188677:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac5180 x1875444570056832/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576180.188685:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576180.188688:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576180.188691:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac5180 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056832:192.168.202.157@tcp:38:kworker.0 00000100:00100000:3.0:1788576180.188696:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056960:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576180.188715:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576180.188726:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576180/real 1788576180] req@ffff920a60ac6300 x1875444570056960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576180.188736:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac6300 x1875444570056960/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576180.188742:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576180.188745:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576180.188748:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac6300 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570056960:192.168.202.157@tcp:8:kworker.0 00000100:00100000:3.0:1788576180.188753:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057088:192.168.202.157@tcp:8:kworker.0 00000800:00000100:3.0:1788576180.188770:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:3.0:1788576180.188779:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576180/real 1788576180] req@ffff920a60ac7100 x1875444570057088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1788576180.188789:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a60ac7100 x1875444570057088/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1788576180.188796:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1788576180.188799:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:3.0:1788576180.188802:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a60ac7100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057088:192.168.202.157@tcp:8:kworker.0 00000100:00100000:1.0:1788576180.189106:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576180.635935:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576181.211716:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576181.662061:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576182.236506:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576182.684305:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1788576183.259809:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576183.707841:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576184.284393:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576184.731914:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576185.307739:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:3.0:1788576185.307775:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576185.307786:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576185.307791:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576185.307804:0:167614:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12938 00000100:00080000:3.0:1788576185.307816:0:167614:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576185.307888:0:167614:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576185.307936:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1c00 x1875444570057216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576185.307967:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576185.307971:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576185.307973:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576185.307978:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12938 00000100:00080000:3.0:1788576185.307981:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576185.307985:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576185.307997:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576185.308011:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2a00 x1875444570057344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576185.308022:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570057216:192.168.202.157@tcp:250:kworker.0 00000100:00080000:3.0:1788576185.308027:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576185.308032:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576185.308034:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576185.308039:0:167614:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12938 00000100:00080000:3.0:1788576185.308043:0:167614:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576185.308048:0:167614:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576185.308054:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576185.308068:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b2d80 x1875444570057472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576185.308078:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576185.308081:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576185.308083:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576185.308087:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12938 00000100:00080000:3.0:1788576185.308098:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576185.308101:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788576185.308107:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:3.0:1788576185.308108:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576185.308120:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3480 x1875444570057600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576185.308152:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576185/real 1788576185] req@ffff920a529b1c00 x1875444570057216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576185.308179:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1c00 x1875444570057216/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576185.308190:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576185.308195:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576185.308198:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570057216:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576185.308225:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057344:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576185.308248:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576185.308260:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576185/real 1788576185] req@ffff920a529b2a00 x1875444570057344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576185.308270:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2a00 x1875444570057344/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576185.308278:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576185.308281:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576185.308284:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2a00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057344:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576185.308289:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057472:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576185.308307:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576185.308318:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576185/real 1788576185] req@ffff920a529b2d80 x1875444570057472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576185.308328:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b2d80 x1875444570057472/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576185.308339:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576185.308342:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576185.308345:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b2d80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057472:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576185.308349:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057600:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576185.308367:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576185.308377:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576185/real 1788576185] req@ffff920a529b3480 x1875444570057600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576185.308388:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3480 x1875444570057600/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576185.308395:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576185.308398:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576185.308401:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057600:192.168.202.157@tcp:8:kworker.0 00000100:00080000:3.0:1788576185.308422:0:167614:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576185.308427:0:167614:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576185.308430:0:167614:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576185.308435:0:167614:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12938 00000100:00080000:3.0:1788576185.308438:0:167614:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576185.308442:0:167614:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576185.308447:0:167614:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576185.308458:0:167614:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0380 x1875444570057728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576185.308697:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057728:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576185.308720:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576185.308731:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576185/real 1788576185] req@ffff920a529b0380 x1875444570057728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576185.308743:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0380 x1875444570057728/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576185.308751:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576185.308754:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576185.308757:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057728:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576185.756523:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576186.331729:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576186.779883:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576187.355784:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576187.804445:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576188.379732:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576188.827868:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576189.410094:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576189.852331:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576190.430387:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576190.431373:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576190.431383:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576190.431388:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576190.431397:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12943 00000100:00080000:1.0:1788576190.431408:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576190.431415:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576190.431465:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893800 x1875444570057856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576190.431497:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576190.431501:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576190.431503:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576190.431507:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12943 00000100:00080000:1.0:1788576190.431511:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576190.431515:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576190.431525:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576190.431543:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890e00 x1875444570057984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576190.431544:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570057856:192.168.202.157@tcp:250:kworker.0 00000100:00080000:1.0:1788576190.431559:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576190.431562:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576190.431564:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576190.431569:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12943 00000100:00080000:1.0:1788576190.431573:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576190.431578:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576190.431582:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576190.431615:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891f80 x1875444570058112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576190.431625:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576190.431628:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576190.431630:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576190.431634:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12943 00000100:00080000:1.0:1788576190.431638:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576190.431643:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000800:00000100:0.0:1788576190.431645:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000008:00100000:1.0:1788576190.431650:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576190.431664:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891500 x1875444570058240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576190.431673:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576190.431676:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576190.431678:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576190.431683:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12943 00000100:00100000:0.0:1788576190.431684:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576190/real 1788576190] req@ffff920a52893800 x1875444570057856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576190.431688:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576190.431696:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576190.431703:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:0.0:1788576190.431711:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893800 x1875444570057856/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576190.431716:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893480 x1875444570058368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1788576190.431727:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576190.431734:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576190.431739:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893800 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570057856:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576190.431766:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057984:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576190.431791:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576190.431817:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576190/real 1788576190] req@ffff920a52890e00 x1875444570057984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576190.431829:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890e00 x1875444570057984/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576190.431837:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576190.431851:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576190.431854:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570057984:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576190.431860:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058112:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576190.431882:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576190.431894:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576190/real 1788576190] req@ffff920a52891f80 x1875444570058112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576190.431905:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891f80 x1875444570058112/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576190.431913:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576190.431917:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576190.431920:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058112:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576190.431925:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058240:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576190.431944:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576190.431956:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576190/real 1788576190] req@ffff920a52891500 x1875444570058240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576190.431966:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891500 x1875444570058240/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576190.431974:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576190.431977:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576190.431980:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891500 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058240:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576190.431985:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058368:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576190.432005:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576190.432016:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576190/real 1788576190] req@ffff920a52893480 x1875444570058368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576190.432033:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893480 x1875444570058368/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576190.432041:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576190.432043:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576190.432046:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058368:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788576190.885254:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576191.453116:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576191.899792:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576192.479134:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576192.926333:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576193.499748:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576193.947872:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576194.526629:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576194.971801:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576195.548869:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00080000:1.0:1788576195.552132:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576195.552142:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1788576195.552146:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576195.552156:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12948 00000100:00080000:1.0:1788576195.552167:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576195.552174:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:1.0:1788576195.552212:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52891880 x1875444570058496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576195.552254:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576195.552258:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1788576195.552259:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576195.552264:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12948 00000100:00080000:1.0:1788576195.552268:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576195.552272:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576195.552282:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576195.552292:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890380 x1875444570058624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576195.552301:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576195.552305:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:1.0:1788576195.552306:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576195.552310:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12948 00000100:00080000:1.0:1788576195.552313:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576195.552316:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576195.552320:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1788576195.552328:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52893b80 x1875444570058752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576195.552337:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576195.552340:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1788576195.552341:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576195.552344:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12948 00000100:00080000:1.0:1788576195.552348:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576195.552351:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576195.552355:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576195.552363:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52890000 x1875444570058880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1788576195.552370:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1788576195.552373:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1788576195.552374:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1788576195.552378:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12948 00000100:00080000:1.0:1788576195.552380:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1788576195.552383:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:1.0:1788576195.552386:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:1.0:1788576195.552395:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a52892680 x1875444570059008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1788576195.560252:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570058496:192.168.202.157@tcp:250:kworker.0 00000800:00000100:0.0:1788576195.560324:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576195.560358:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576195/real 1788576195] req@ffff920a52891880 x1875444570058496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576195.560375:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52891880 x1875444570058496/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576195.560386:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576195.560391:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576195.560395:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52891880 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570058496:192.168.202.157@tcp:250:kworker.0 00000100:00100000:0.0:1788576195.560402:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058624:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576195.560422:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576195.560432:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576195/real 1788576195] req@ffff920a52890380 x1875444570058624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576195.560450:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890380 x1875444570058624/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576195.560458:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576195.560461:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576195.560465:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890380 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058624:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576195.560471:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058752:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576195.560487:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576195.560496:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576195/real 1788576195] req@ffff920a52893b80 x1875444570058752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576195.560507:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52893b80 x1875444570058752/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576195.560515:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576195.560518:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576195.560522:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52893b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058752:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576195.560528:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058880:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576195.560544:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576195.560552:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576195/real 1788576195] req@ffff920a52890000 x1875444570058880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576195.560563:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52890000 x1875444570058880/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576195.560576:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576195.560580:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576195.560583:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52890000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570058880:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576195.560616:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059008:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576195.560635:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576195.560645:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576195/real 1788576195] req@ffff920a52892680 x1875444570059008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576195.560654:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a52892680 x1875444570059008/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576195.560661:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576195.560664:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576195.560666:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a52892680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059008:192.168.202.157@tcp:8:kworker.0 00010000:00010000:1.0:1788576195.996194:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576196.571793:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576197.019953:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576197.595807:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576198.046893:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576198.621699:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576199.068067:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1788576199.643787:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576200.091891:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1788576200.669866:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) f79285ed-4685-45a8-8731-5e2522a143f3->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576200.669878:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1788576200.669883:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576200.669893:0:155040:0:(import.c:545:import_select_connection()) MGC192.168.202.157@tcp: connect to NID 192.168.202.157@tcp last attempt 12954 00000100:00080000:3.0:1788576200.669903:0:155040:0:(import.c:630:import_select_connection()) MGC192.168.202.157@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576200.669911:0:155040:0:(import.c:665:import_select_connection()) MGC192.168.202.157@tcp: import ffff920a54164800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000100:00100000:3.0:1788576200.669946:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0000 x1875444570059136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576200.670311:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570059136:192.168.202.157@tcp:250:kworker.0 00000800:00000100:1.0:1788576200.670374:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00080000:3.0:1788576200.670434:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576200.670438:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1788576200.670440:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576200.670445:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12954 00000100:00080000:3.0:1788576200.670449:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576200.670453:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0000-mdc-ffff920a47b93800: import ffff920a4549f800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576200.670464:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576200.670478:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b0700 x1875444570059264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1788576200.670531:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576200/real 1788576200] req@ffff920a529b0000 x1875444570059136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576200.670548:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0000 x1875444570059136/t0(0) o250->MGC192.168.202.157@tcp@192.168.202.157@tcp:26/25 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576200.670564:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a54164800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576200.670567:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of MGS on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576200.670571:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:f79285ed-4685-45a8-8731-5e2522a143f3:2396:1875444570059136:192.168.202.157@tcp:250:kworker.0 00000100:00100000:1.0:1788576200.670605:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059264:192.168.202.157@tcp:38:kworker.0 00000800:00000100:1.0:1788576200.670627:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:1.0:1788576200.670638:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576200/real 1788576200] req@ffff920a529b0700 x1875444570059264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1788576200.670649:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b0700 x1875444570059264/t0(0) o38->lustre-MDT0000-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1788576200.670656:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a4549f800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1788576200.670659:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:1.0:1788576200.670662:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b0700 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059264:192.168.202.157@tcp:38:kworker.0 00000100:00080000:3.0:1788576200.670912:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-MDT0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576200.670917:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0001_UUID: starting recovery 00000100:00080000:3.0:1788576200.670919:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576200.670924:0:155040:0:(import.c:545:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12954 00000100:00080000:3.0:1788576200.670928:0:155040:0:(import.c:630:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576200.670931:0:155040:0:(import.c:665:import_select_connection()) lustre-MDT0001-mdc-ffff920a47b93800: import ffff920a51ac4800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576200.670938:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683394 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1788576200.670951:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3100 x1875444570059392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576200.670961:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576200.670965:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1788576200.670966:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576200.670969:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12954 00000100:00080000:3.0:1788576200.670972:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576200.670975:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0000-osc-ffff920a47b93800: import ffff920a51ac1800 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576200.670979:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576200.670987:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b1f80 x1875444570059520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1788576200.670994:0:155040:0:(pinger.c:229:ptlrpc_pinger_process_import()) 02515dca-bbf0-45e0-bd6e-e76f0c89bd0b->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1788576200.670996:0:155040:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1788576200.670998:0:155040:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1788576200.671001:0:155040:0:(import.c:545:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: connect to NID 192.168.202.157@tcp last attempt 12954 00000100:00080000:3.0:1788576200.671004:0:155040:0:(import.c:630:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1788576200.671006:0:155040:0:(import.c:665:import_select_connection()) lustre-OST0001-osc-ffff920a47b93800: import ffff920a51ac1000 using connection 192.168.202.157@tcp/192.168.202.157@tcp 00000008:00100000:3.0:1788576200.671009:0:155040:0:(osc_request.c:3830:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683394 ocd_grant: 8388608, lost: 0. 00000100:00100000:3.0:1788576200.671016:0:155040:0:(import.c:870:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff920a529b3b80 x1875444570059648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1788576200.671075:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00000100:00100000:0.0:1788576200.671209:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059392:192.168.202.157@tcp:38:kworker.0 00000800:00000100:0.0:1788576200.671238:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576200.671251:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576200/real 1788576200] req@ffff920a529b3100 x1875444570059392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576200.671263:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3100 x1875444570059392/t0(0) o38->lustre-MDT0001-mdc-ffff920a47b93800@192.168.202.157@tcp:12/10 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576200.671270:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac4800 lustre-MDT0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576200.671274:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-MDT0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576200.671277:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3100 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059392:192.168.202.157@tcp:38:kworker.0 00000100:00100000:0.0:1788576200.671283:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059520:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576200.671299:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576200.671307:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576200/real 1788576200] req@ffff920a529b1f80 x1875444570059520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576200.671317:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b1f80 x1875444570059520/t0(0) o8->lustre-OST0000-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576200.671326:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576200.671328:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576200.671331:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b1f80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059520:192.168.202.157@tcp:8:kworker.0 00000100:00100000:0.0:1788576200.671337:0:2396:0:(client.c:1920:ptlrpc_send_new_req()) Sending RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059648:192.168.202.157@tcp:8:kworker.0 00000800:00000100:0.0:1788576200.671351:0:2396:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.202.157@tcp 00000100:00100000:0.0:1788576200.671359:0:2396:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1788576200/real 1788576200] req@ffff920a529b3b80 x1875444570059648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1788576200.671369:0:2396:0:(client.c:2535:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff920a529b3b80 x1875444570059648/t0(0) o8->lustre-OST0001-osc-ffff920a47b93800@192.168.202.157@tcp:28/4 lens 520/544 e 0 to 1 dl 1788576225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1788576200.671376:0:2396:0:(import.c:65:import_set_state_nolock()) ffff920a51ac1000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1788576200.671379:0:2396:0:(import.c:1555:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.202.157@tcp failed (-110) 00000100:00100000:0.0:1788576200.671383:0:2396:0:(client.c:2415:ptlrpc_check_set()) Completed RPC req@ffff920a529b3b80 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:02515dca-bbf0-45e0-bd6e-e76f0c89bd0b:2396:1875444570059648:192.168.202.157@tcp:8:kworker.0 00010000:00010000:0.0:1788576201.115831:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576201.692664:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576202.140742:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576202.715941:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:0.0:1788576203.183340:0:160930:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576203.744275:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds 00010000:00010000:1.0:1788576204.187847:0:160929:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1788576204.764671:0:167275:0:(client.c:2716:ptlrpc_set_wait()) set ffff920a51abe6c0 going to sleep for 0 seconds Debug log: 79120 lines, 79120 kept, 0 dropped, 0 bad.